(NASA-CR-185444) AN AUTONATION SIMULATION N89-260 17 

TESTBED Report, Oct. ,1987 - Sep. 1983 
(Vanderbilt Univ. ) 114 p CSCL 14B 

Unclas 
G3/09 0219561 


AN AUTOMATION SIMULATION TESTBED 


Phase I Report 

October 1987 - September 1988 


National Aeronautics and Space Administration 
Marshall Space Flight Center 


Produced by 
Vanderbilt University 
Department of Electrical Engineering 

Authors: 

Dr George E. Cook 
Dr Janos Sztipanovits 
Dr Csaba Biegl 
Dr Gabor Karsai 
James F. Springfield 
Atheel Mutammara 



Contracting Officer Representative: Dr Kenneth R. Fernandez 


Grant No. NAG-8690 


TABLE OF CONTENTS 



EXECUTIVE SUMMARY 1 

1. INTRODUCTION 5 

• 2. WORKSTATION IMPLEMENTATION OF ROBOSIM 6 

2.1 The HP350SRX Graphics Workstation 6 

2.2 The MD Program 10 

2.3 The R2 Program 11 

® 2.4 Simulation Library and Environment 28 

2.5 Inverse Kinematics 29 

2.6 Collision Detection 31 

2.7 Surgical Positioner 38 

^ A-2.1 APPENDIX: Structure Declarations for the Simulation Library 41 

A-2.2 APPENDIX: Simulation Library Functions 44 

3. INTELLIGENT GRAPHICS MODELING ENVIRONMENT 47 

• 3.1 Critique of the Current Graphics Modeling Technique 47 

3.2 System Design of the Graphic Simulation Environment 48 

3.3 Detailed Description of the Components 51 

3.4 Automation Interface for Structural Modeling Systems 60 

• 4. CASE STUDIES 73 

4.1 Space Station Modeling Using ROBOSIM 73 

4.2 Operational Modeling of the Space Station 75 

9 4.3 Study of the Space Station ECLSS 84 

5. Future Work 112 


1 



EXECUTIVE SUMMARY 


The objective of this research program has been twofold. First, the basic capabilities 
of ROBOSIM (a graphical simulation system developed jointly by NASA-MSFC and 
Vanderbilt University) are being improved and extended by taking advantage of advanced 
graphic workstation technology and artificial intelligence (AI) programming techniques. 
Second, the scope of the graphic simulation testbed is being extended to include general 
problems of Space Station automation. 

The first objective is a logical continuation of the joint NAS A/ Vanderbilt ROBOSIM 
development. State-of-the-art graphic workstations offer new opportunities for simulation 
of complex, linked geometrical structures. Hardware support for 3-D graphics and high 
processing performance make high resolution solid modeling, collision detection, and 
simulation of structural dynamics computationally feasible. With the introduction of new 
AI programming techniques, graphic structural simulation can be combined with high-level, 
Al-based control functions; thus the simulation testbed can support studies in task level 
planning and in other issues of autonomous control. 

The Space Station is a vastly complex system with many interacting subsystems. 
Automation, being a decisive factor in crew productivity and safety, is expected to play a 
major role in the Space Station operation. The rationale for the second objective of this 
project is based on the fact that formulation and testing of automation concepts require 
understanding the behavior of and the interactions among the various subsystems. For 
example, the Environmental Control and Life Support System (ECLSS), which is one of 
the most complex subsystems in the Space Station, is an aggregate of interdependent 
mechanical, chemical and electrical processes. These processes interact with each other 
and impose constraints on the operation of other subsystems in many levels. The following 
list includes a few examples for these interactions: 

1. the air temperature control in the ECLSS is directly related to the Thermal subsys- 
tem. 



2. the ECLSS is one of the major electric energy consumers in the Space Station, 
therefore its operation interacts with the Electric Power Supply subsystem, 

3. effects of the ECLSS operation on the utility consumers (air, potable water, hygienic 
water, wash water, etc.), 

4. waste material removal may interact with low-gravity experiments. 

Design and testing of automation concepts demand modeling of the affected 
processes, their interactions and that of the proposed control systems. These models may 
vary in objective and sophistication, in accordance with the level of control functions to be 
studied. The analysis of elementary control loops that maintain the value of a process 
variable require the use of high fidelity dynamic simulation. The testing and validation of 
higher level and autonomous controllers will necessitate the use of Al-based models 
representing qualitative as well as quantitative features of processes. Extended modeling 
techniques include the explicit description of hierarchical process structures, causal rela- 
tions, fault propagation models and component hierarchies. 

The automation testbed has been designed to facilitate studies in Space Station 
automation concepts. Its main purpose is to provide cost-effective solutions for the analysis 
of the interactions among work packages, and for experiments with the scars and hooks 
provided by the IOC automation concepts for advanced automation. Supplementing the 
ROBOSIM graphical simulation package with the required new capabilities is a complex 
task. It requires significant extension of the system in many ways, including the incorpora- 
tion of Al-based modeling tools, the application of automatic program generation facilities 
for fast prototyping, and the introduction of advanced software engineering techniques for 
managing large-scale models. 

In this Report, the first steps of this process are discussed. In the first section the new 
capabilities of the graphic workstation version of ROBOSIM are described. The work 
accomplished in the first year of the project has resulted in significantly improved 3-D 
graphic capabilities, interactive model building tools, and a solution for collision detection. 
The second section discusses the design details of the next version of the graphical simula- 
tion package. The new design makes it possible to integrate the system with tools support- 
ing automation studies as well. The third section provides case studies that demonstrate the 
usage and capabilities of an integrated structural modeling and automation testbed. The 
case studies include the structural model of the Space Station, a study of the interactions 
between the attitude control and electrical energy supply system, and a simplified process 



and failure model of the ECLSS subsystem. The fourth section describes the work plan for 
the second year effort and presents a work schedule to track the work proposed. 

The work described in this paper has been mostly performed on a Hewlett Packard 
9000/350 SRX graphics workstation. We would like to express our gratitude to the In- 
dustrual Application Center of the Hewlett Packard Company for their support which 
made this research possible. 



1. INTRODUCTION 


This report is organized into five chapters. Chapter 2 which follows the introduction 
describes the work done in porting ROBOSIM to the HP350SRX graphics workstation. 
New additional ROBOSIM features, like collision detection and new kinematics simulation 
methods are also discussed here. 

Chapter 3 can be divided into two parts. In the first part - based on the experiences of 
the work on ROBOSIM - we suggest a new graphics structural modeling environment, 
which is intended to be a part of a new knowledge-based multiple aspect modeling testbed. 
The second part of the chapter contains a description of the knowledge-based modeling 
methodologies and tools already available to us. 

Chapter 4 contains three case studies in the area of Space Station automation. First a 
geometrical structural model of the station is presented. This model was developed using 
the ROBOSIM package. Next the possible application areas of an integrated modeling 
environment in the testing of different Space Station operations are discussed. One of these 
possible application areas is the modeling of the Environmental Control and Life Support 
System (ECLSS), which is one of the most complex subsystems of the station. Using the 
multiple aspect modeling methodology presented in Chapter 3 we are building a fault 
propagation model of this system, which is described at the end of the chapter. 

Chapter 5 concludes the report by suggesting possible future research directions for 
the application of these modeling techniques in automation systems. 
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2. WORKSTATION IMPLEMENTATION OF ROBOSIM 


This chapter describes the work which has been done in order to enhance the 
capabilities of the ROBOSIM graphical structure modeling package. ROBOSIM in its 
original form was a command-oriented modeling language, with a not too user friendly 
programming interface. Furthermore its graphics capabilities were limited, due to the fact 
that originally it was designed for use on a remote graphics terminal attached to a VAX- 
like processor, which did not offer many of the features available on modern graphics 
engineering workstations. Further additions include simulation libraries for collision 
detection and dynamics, which are also described later in this chapter. 


2.1 The HP350SRX Graphics Workstation 


Since part of the impetus for extending ROBOSIM was the capabilities provided by 
graphics workstations, it is necessary to understand these capabilities. All of the informa- 
tion that follows is specifically oriented towards the HP350SRX workstation; however, 
much is generally applicable to other workstations. 

The HP350SRX workstation has a pixel resolution of 1280x1024 pixels. There are 16 
image planes and 4 overlay planes. Each plane is one bit per pixel. Typically the image 
planes are used for graphics and the overlay planes are used for XWindows. When the 
image planes are rapidly changed (animation) flickering results if the images are not 
double buffered. This means that only 8 image planes are actually available. While one set 
of image planes are being displayed, the other set is being changed. Then, the sets are 
switched. This results in flicker-free motion at the cost of reduced numbers of colors. Since 
only eight planes are used at one time, only 256 colors can be displayed at one time. The 
overlay planes, if used, will hide the image planes. Therefore, if X is being used, a 
transparent window is created. This allows X applications to be run while seeing what is in 
the image planes. 



The most important capability of the workstation is the increased speed and facilities 
provided by the Starbase graphics library and the hardware graphics accelerator. These 
facilities allow display of three dimensional graphics objects with options such as hidden 
surface removal, shading, perspective views, and colors. 

The hardware accelerator includes a matrix multiplier. This allows multiplication of 
4x4 matrices much faster than could be done in software. This facility is used to a great 
extent in display of objects. There are many coordinate transformations occurring during 
display such as rotation and translation of objects in modeling coordinates, conversion of 
modeling to world coordinates, perspective transformations, world to virtual device coor- 
dinates, and virtual device to device coordinates. Each of these involves multiplying by 
matrices; also, there can be many levels of transformations in modeling coordinates. All 
graphic objects are "put through the pipeline" of transformations, and the hardware multi- 
plier is a key part in providing real-time speed. There is one difference between the trans- 
formation matrices used in Starbase and the ones traditionally used by roboticists. The 
graphics standard uses matrices that are the transpose of the ones used in robotics. There- 
fore, all matrices in the programs are represented in the graphics standard form. For this 
reason, all matrix equations had to be the reverse of those used in robotics. 

Another useful feature of Starbase is the display list. A display list is made up of 
segments. Each segment can be thought of as a procedure. One segment can call another 
and the called segment returns to the calling segment when finished. Almost any Starbase 
function can be placed in a segment. Then, whenever that segment is traversed those 
commands are executed. This is very useful; for instance, all of the Starbase polygon 
procedure calls that make up a robot link can be placed in a segment with a transformation 
matrix that represents the transformation resulting from a particular value for that link’s 
joint variable. Then, changing the transformation matrix in the display list will result in that 
link "moving" the next time that display list is traversed. A segment network is shown 
below. It has been printed from the simulation program for an actual robot. It has been 
abbreviated in parts, "fd" is the file descriptor returned by Starbase when a display is 
opened for graphic output. The {} indicate an array that has not been printed out. Segment 
#0 is the main segment. It has a call to segment #1. Segment #1 is for a robot. If there 
were another robot, then there would be another call in segment #0. Segment #1 first 
pushes a matrix onto the transformation stack. This transformation corresponds to the 
position of the robot in the world. Next, segments 2-9 are called. In segment #2 the first 
concat_transformation3d is a transformation describing the structure of the link. The 
second transformation describes the current value of the joint variable. Concat multiplies 
the matrix by whatever is currently on the transformation stack and pushes the result back 



on the stack. Now, the polygons in segment #2 are displayed after first being transformed 
by whatever is on top of the transformation stack. Segment #2 then returns control to 
segment #1, and traversal continues through segment #9. When segment #9 returns, the 
top of the matrix is popped off and returned to the state it was in before traversal began. 

segment 0 begin 
move3d( fd, 0, 0, 0) 
dl_label( fd, 1) 
call_segment( fd, 1) 
segment 0 end 

segment 1 begin 

push_matrix3d( fd, {}) 
call_segment( fd, 2) 
call_segment( fd, 3) 
call_segment( fd, 4) 
call_segment( fd, 5) 
call_segment( fd, 6) 
call_segment( fd, 7) 
call_segment( fd, 8) 
call_segment( fd, 9) 
pop_matrix( fd) 
segment 1 end 

segment 2 begin 

concat_transformation3d( fd, {}, 0, 0) 
concat_transformation3d( fd, { }, 0, 0) 
polygon3d( fd, {}, 5, 1) 


polygon3d( fd, {}, 5, 1) 
segment 2 end 


segment 9 begin 

concat_transformation3d( fd, {}, 0, 0) 
concat_transformation3d( fd, {}, 0, 0) 
polygon3d( fd, {}, 5, 1) 


polygon3d( fd, {}, 5, 1) 
segment 9 end 

The ability to pick an object that is displayed on the screen is a very important part in 
the graphics editor. Starbase provides a simple way to do this. When a display list is dis- 
played on the screen the points making up an object are eventually converted to actual 
device coordinates. Now, given a range of coordinates, Starbase can return information 



regarding what is displayed in that range. This consists of the segment number, the most 
recent label within that segment (if any), and the offset from that label. For instance, if the 
first polygon in segment #9 fell within that window, then Starbase would return segment 
#9, label #0 (there is no label in segment #9), and an offset of 3 (the first polygon is the 
third command in segment #9). Starbase can even return the entire path through the 
display list, giving all called segments and offsets leading up to the polygon in segment #9. 

XWindows provides the ability to read the mouse position. A program can read the 
position of the mouse and convert that position to the form required by Starbase. Thus, one 
can use the mouse to point to an object on the screen, and a program can figure out what is 
being pointed to. 

XWindows also provides many other facilities that proved to be useful in implement- 
ing this work. One of the most useful aspects of X is the menus. Using X, one can imple- 
ment menus very easily. This allows user-friendly interfaces to be written without having to 
deal with the complexities introduced. For instance, a set of menus can be created to 
manipulate some display list. The menu entries are created and X is told which procedures 
to execute upon selection of the corresponding menu entry. A transparent window in the 
center of the screen allows the image planes (graphics planes) to be seen through the X 
application. 

There are also two other peripherals which have been extensively used. The button 
box and the knob box provide very easily used input capabilities. Once the devices have 
been opened for use by a program it is quite simple to poll them. The button box returns an 
integer corresponding to the button pushed, if any. The knob box is just as simple to poll, 
but it has additional features. The knob box has nine knobs and each can be set differently. 
A knob’s range can be set; for example, a knob can return a number between -1. and 1. or it 
can return a number between 10. and 100. Also, the knob can be preset to a particular 
value. This means that whatever position the knob is in, that position corresponds to the set 
value. 

The features of the HP350SRX workstation make it ideal for use in high-performance 
graphics applications. The resolution and color capability allow for sophisticated graphics. 
The graphics accelerator provides speed, and the display lists provide easy access to 
graphics hardware. XWindows allows friendly and generic user interfaces to be written 
simply and easily. And the peripherals such as the mouse, button box, and knob box 
provide a flexible and diverse range of input. 



2,2 The MD Program 


Porting ROBOSIM to the HP350SRX workstation added no additional features to 
those found in the VAX version. ROBOSIM on the HP no longer used the TEKTRONIX 
4014 interface, although XWindows allowed certain windows to operate in a TEKTRONIX 
emulator mode. ROBOSIM was adapted to use the Starbase graphics move and draw 
commands. ROBOSIM still performed all transformations internally, but used a window 
from X and Starbase graphics for output. This allowed one window in which to run the 
process and another in which to see the output. This capability spurred an early attempt at 
allowing an interactive mode of operation in which ROBOSIM commands were typed in, 
and the effects were immediately seen in the display window. However, this method was 
never effectively implemented or used. 

The MD program originally evolved as a means of displaying a robot that had been 
generated by ROBOSIM. Through this program, a user could display a robot and set colors 
and other attributes such as hidden surface removal, shading, and specular reflection. Also, 
the camera position (i.e. the position from which the object is looked at) could be changed 
to provide views of the object from many different perspectives. 

Extensions to the basic MD allowed multiple robots and objects, and it even has 
provisions to accept joint angles and other parameters from a separate process. With this 
feature, a primitive simulation can be run. An early use of this involved a lisp process 
piping commands to a space station model that would orient the solar collectors to receive 
maximum exposure. MD was also able to run in a mode in which joint angles were read 
from a file and the robot’s joints were cycled through these. This feature was used for 
simple simulations of downhand welding. Two robots, one a six degree of freedom robot, 
and the other a two degree of freedom positioner, were simulated. The robot performed 
the welding and the positioner assisted in maintaining the downhand position and proper 
orientation of the wire feed to the direction of movement of the torch. The joint angles 
were generated by a separate program and stored in two files. Using MD, one could look at 
the robots from various positions to visually verify that the downhand welding was working 
correctly. 

The basic structure of MD involves loading the link files of a robot and creating 
display segments corresponding to each link. Each segment has a transformation matrix 
and a polygon list. Also, there is a segment for the entire robot that has a transformation 
matrix describing the position of the robot in the world, and commands that set the color 



and other parameters for the robot. The input devices for MD are the button box and the 
knob box. These devices provide the ability to turn functions of Starbase on and off and to 
adjust parameters of Starbase. For most functions, there is a one-to-one correspondence 
between Starbase functions and MD functions. MD is useful for looking at a robot after it 
has been made by ROBOSIM. The robot can be brought up on the screen, looked at from 
various positions, and the joints can be moved. 

The capabilities of MD for more complicated simulation were very limited, and 
further work on MD was replaced by the development of the simulation library and environ- 
ment. MD is still used for photographing robots and other structures such as the space 
station. It is also still used for quickly verifying robots or other structures that have been 
constructed with ROBOSIM. Although it is not used directly for simulation purposes now, 
the components of it dealing with graphics manipulations are still used in R2 and other 
programs. 


2.3 The R2 Program 


The development of R2 arose from the capabilities provided by MD and the need for 
an easier to use and more flexible interface to ROBOSIM. R2 was designed to overcome 
some of the limitations of ROBOSIM while taking advantage of the facilities available on 
graphics workstations. However, complete compatibility with ROBOSIM was desired; this 
was accomplished by the output of R2 being ROBOSIM code. Having R2 generate 
ROBOSIM code allowed R2 to be much simpler. It was not necessary to reimplement what 
ROBOSIM already provided. This method has proved to be the most flexible. Now, robots 
can be designed by writing a ROBOSIM program, using R2 to generate a ROBOSIM 
program, directly generating files from custom programs, or any combination thereof. 

ROBOSIM provides a simple way in which to design robots. Based on the specifica- 
tions in a user-written ’program’ a file for each link is generated. This file contains the 
vector list that is used to draw the robot, the A-matrix, the Denavit-Hartenburg 
parameters, joint types, and the pseudo-inertia matrix. However, this method requires the 
user to maintain a lot of information that the computer can handle much more easily. Since 
ROBOSIM creates every object at the origin, the user must keep track of each objects’ 
dimensions in order to place it such that it will be in the proper position and orientation 
with respect to the other objects in a link. The only other method that ROBOSIM allows is 
to load in data files that have been generated by some other method. This requires a 



custom written FORTRAN program with appropriate calls to ROBOSIM functions. This is 
the most flexible way in which to use ROBOSIM, but also the most difficult. What is 
needed is a flexible, but user-friendly, environment in which to design robots. 

Before discussing the internals of R2, it is useful to see how it works from a user’s 
point of view. What follows is basically a user’s manual for R2. However, some knowledge 
of ROBOSIM is expected. For information see the ROBOSIM manual and tutorial. It is 
recommended to read the following while running R2. Proper execution of all capabilities 
requires the proper setup of several files and directories. This is explained in the 
ROBOSIM manual. Execute R2 from your ’source’ directory. 

First, R2 is designed to run under XWindows. Therefore, type xstart to run XWin- 
dows. To execute the program type r2 [-t terminal] [-m message_level]. The default ter- 
minal type is "hp98721". The only other terminal currently recognized is a ”hp300h". The 
message_level refers to the amount of help that is available. The default level is level 0. At 
this level only error messages are displayed. At level 1, a small window is created in the 
upper left comer. Then, whenever the program is waiting for input from the user, an 
appropriate message is displayed. Level 2 is the highest level; after any menu item is 
selected, a window with information describing the command is displayed. When the 
information has been read, the user clicks the mouse on the "OK" button. The user inter- 
face consists of the graphics window, where the model is displayed, a line of menus across 
the top, a diagram showing the current meaning of the buttons, and a diagram of the knobs 
showing their meaning. The use of the button box and knob box in R2 is the same as that in 
MD. 

MOUSE: R2 is designed to make extensive use of the mouse. The only time at which the 
user uses the keyboard is when it would be more difficult to use the mouse. This only 
occurs when requesting a file name for the robot, or environment. At all other times, 
input is received from the mouse, the button box, or the knob box. To select a menu 
move the mouse’s cursor until the desired menu heading is highlighted. Now, press 
either of the mouse’s buttons and hold it. The menu will appear below. While holding 
the button down, move the cursor down the menu until the desired menu entry is 
highlighted; then release the button. If (before releasing the button) you decide that 
the wrong menu has been selected, move the cursor out of the menu and release the 
button. If you have already selected a menu item, most functions provide a means to 
cancel them with no effects. 


Numeric Input Window: Many functions make use of this window. It consists of the num- 


bers 0-9, a decimal point, a minus sign, CANCEL, END, and a set of parameters 
(such as radius and height for a cylinder, or X,Y, and Z for translate). When invoked, 
all parameters are initialized to zero. However, all objects that are made must have 
positive values. To select a parameter move the mouse cursor over the desired 
parameter and press the LEFT button on the mouse. Then use the mouse to enter the 
desired value. If you make a mistake, simply press the parameter "button" again, 
which will set the parameter to zero and allow you to reenter that parameter. When 
finished entering parameters, select END. If all is well, the command will be ex- 
ecuted. If at any point you decide to abort this command, then press the CANCEL 
button in the window. 

QUITMENU: 

Quit: exits from the program. 

Restart: deletes the current model from memory, but does not exit the program. 

MAKE OBJECT MENU: 

Box: uses the numeric input window (described above). This command has three 
parameters: X, Y, and Z. These three parameters are the dimensions of the box 
along the three coordinate axes. 

Cylinder: uses the numeric input window (described above). This command has two 
parameters: radius and height. The cylinder is created with height along the z- 
axis. 

Cone: uses the numeric input window (described above). This command has two 
parameters: radius and height. 

Truncated Cone: uses the numeric input window (described above). This command 
has three parameters: upper radius, lower radius, and height. 

Sphere: uses the numeric input window (described above). This command has one 
parameter, the radius of the sphere. 


Special Surface: This command is used for creating custom objects. You do this by 
first creating a polygon and then extruding or revolving it to create a solid 



object. The right button selects the starting point. The left button draws a line. 
To adjust the scale push the scale button, and enter a value at least two times 
the amount of your largest coordinate. The resolution is useful for specifying the 
smallest unit that will be differentiated. If every point is a multiple of five, then 
set the resolution to five. (SPECIAL NOTE: you must define the polygon in a 
counterclockwise direction for extrude and clockwise for revolve.) WARNING: 
DUE TO IMPLEMENTATION CONSTRAINTS IN THE SIMULATOR’S 
COLLISION DETECTION ALGORITHM, ALL POLYGONS MUST BE 
CONVEX. AT THIS TIME NO CORRECTION OR DETECTION OF CON- 
CAVE POLYGONS IS MADE, SO IT IS THE RESPONSIBILITY OF THE 
USER TO PROVIDE THIS CHECK. 

Clone: allows the copying of an object. This is especially useful for copying the 
custom designed objects since they require the most work. After selecting clone, 
select the object to be cloned, with the mouse. 

MANIP OBJECT MENU: 

Translate: uses numeric input window (see description above). This command has X, 
Y, Z, and HOME for parameters. Translations are relative (i.e. they occur 
relative to the current position). To return an object to its home position, press 
HOME, T H , and END. 

Rotate absolute: uses numeric input window. However, the X,Y, and Z here represent 
rotations around the corresponding axes. Rotations are absolute, not additive. If 
you specify a rotation on an object, and then later another rotation, the first 
rotation is lost and the new rotation is from the objects’ home position. 

Rotate relative: same as rotate above, except that these rotations are from the current 
position. 

Delete: waits for you to select an object for deletion. Use the mouse to select the 
object. Pressing the left button of the mouse while not on an object cancels this 
command. 

Attach: lets one object be attached to another object. First, use the mouse to select 
the base object, then select the polygon of the base object where the attachment 
is to be. Then, select the object to be attached and finally the polygon of the 



attached object. This command will attach the two objects selected such that the 
two polygons selected line up. This attachment creates a hierarchy such that the 
movement of the base object occurs to the attached object, but a movement of 
the attached object will not affect the base object. The new home position of the 
attached object is its position as attached to the base object. Once an object is 
attached it can not be unattached. The object must be deleted and made again. 

Resize: lets an object be resized. It is especially useful along with the attach function. 
If several objects are created and attached together, then any of them can be 
resized and the relationship between them will be maintained. After selecting 
resize, the object to be resized is selected with the mouse. Then a window 
identical to the one used to create it appears. Enter the new dimensions, select 
END and the object will be resized. 

LINKS MENU: 

Revolute Joint, Prismatic Joint, Fixed Joint: These three commands create a joint of 
the corresponding type. After selecting an entry the user is prompted to select 
whether it is to be an I-Joint or an 1+ 1-Joint. An I-Joint is the place of attach- 
ment to the previous link, and an I+l-Joint is the place of attachment to the 
next link. 

Rotate, Translate, Delete, Attach: These commands operate just like the ones in the 
MANIP OBJECT MENU. The reason to have separate commands for joints is 
that it is difficult to select them on the screen with the mouse. 

Check joints for validity: This command checks the relationship between the I and the 
1+1 joint to make sure that it follows the Denavit-Hartenburg convention, as 
required by ROBOSIM. 

FILE MANAGEMENT MENU: This menu provides three basic capabilities: save a ses- 
sion, load a session, generate ROBOSIM code, and run MD. 

Save file: This command saves the current model. The user is prompted as to whether 
it is to be saved as an environment file, a link file, or to exit this command. Then 
the user is prompted for a robot name and then for an extension. 

Load file: This command loads a previously saved model. The user is prompted in the 
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same way as save file above. 


Generate ROBOSIM File: This command prompts first as to whether the file to be 
generated is for a robot or environment. Then the name is asked for. The 
ROBOSIM file is then generated, ROBOSIM is called and the file is executed. 
Control is then passed back to R2. The robot or environment can now be 
viewed by MD, if R2 is running on an hp98721 display. 

MD: This command executes the MD program. This allows the robot to be viewed 
completely. Does not work with environment files presently. Also, can not be 
executed on an hp300h. 

HP300 MENU: This menu implements some functions on the hp300h. Since this machine 

does not have the button box or knob box it is necessary to implement them this way. 

Look From: This command uses the numeric input window. Specify the X, Y, and Z 
coordinates to look from. At least one must be non-zero. 

Look At: This command uses the numeric input window. Specify the X, Y, and Z 
coordinates to look at. 

Although the interface gives the appearance of an object oriented structure, it is not 
implemented in this manner. The basic structure in this program is an array of pointers. 
Currently, this is set to a size limit of 100. This means that the most objects that can be in 
one link is 100 primitives. However, this number can be set to anything and the program 
recompiled. A better structure would be a linked list of objects that is dynamically allo- 
cated. At present, however, this method has not been a problem. The actual C structure 
declarations are shown below. 

#defme MAXOBS 100 
#define MAXKIDS 10 

typedef struct vertex { 
float x; 
float y, 
float z; 
float md; 

} vertextype; 

typedef struct { 

int s_p; /‘source polygon*/ 

int d_o; /‘destination object*/ 



int d_p; /^destination polygon*/ 

} childtype; 

typedef struct object { 
char ‘name; 
int type; 
int vertices; 
vertextype *model; 
int custom_vertices; 
float custom_extrude; 
vertextype *custom_model; 
float size[5]; 
float amat[4][4]; 
float ref[4][4]; 
int display list; 
childtype kids[MAXKIDS]; 

} objecttype; 

Whenever an object is created, enough memory for a structure of type objecttype is 
allocated and the pointer to this memory is stored in the array. All information relating to a 
particular object is stored in this structure. The first element in this structure is the name of 
the object. This name is actually the ROBOSIM command that is used to generate this 
object (i.e. BOX, R-JOINT-I). The name also directly corresponds to the next element: the 
type. The type is an integer that represents the ROBOSIM command. The variable 
’vertices’ is the number of points in the model. The variable ’model’ is a pointer to the list 
of vertices that describe the graphic model. The ’custom_vertices’ is the number of points 
in the polygon that is used to generate a custom surface (REV-SURFACE and 
EXTRUDE-SURFACE). ’Custom_extrude’ is used in an extrude-surface object; it is the 
amount the object is extruded. The ’customjnodel’ is a pointer to the polygon that is used 
in a custom surface. The ’size’ array is an array of parameters that can be used as the 
arguments to primitive calls. For instance, if the object is a box, then the first three ele- 
ments of ’size’ will be used to store the x,y, and z dimensions. The ’amat’ variable is a 
matrix representing a transformation (rotation and translation) on the object. The ’ref 
variable is also a transformation, but it is used to define the home position of the object. 
The ’display_list’ variable is an integer that is the descriptor of the display_list in which this 
object is stored. The display_list is a set of graphics functions that when traversed will result 
in the graphic object being displayed. The ’kids’ array is an array describing the children of 
an object. One object becomes another object’s child when the child object is attached to 
the parent object. Currently, the maximum number of children one object can have is ten. 
However, this value can be changed. Each child is described by three integers. The first, 
’s_p’ or source polygon is the number of the polygon of the parent where the child is at- 
tached. The ’d_o’ or destination object is the array index of the object that is attached. The 
’d_p’ or destination polygon is the polygon of the child object that is attached to the parent. 


After an object is selected from the menus and the parameters have been entered, the 
object is created. The FORTRAN code that generates the primitives in ROBOSIM is also 
used in R2. The use of the same code ensures that what is seen in the editor is the same as 
what will be by ROBOSIM. The FORTRAN routines store the vector lists in an array that 
is passed to them. After getting this information, the editor stores it in the structure allo- 
cated for the object and in a slightly different form in a display list. The other variables in 
the structure are filled out, the transformation matrices are set to identity, and a call to the 
newly created display list is inserted into the root display list. Now, the next time the 
displayjist is traversed the object will be displayed. 

Once an object has been created (i.e. an instance is made of the object), ’messages’ 
can be sent to it. From the user’s point of view, this is what is done. However, the im- 
plementation is different. The object is selected by picking it with the mouse. R2 waits for a 
mouse button to be pressed and then reads the (x,y) location of the mouse. These coor- 
dinates are then used by Starbase to determine what primitive is in that area. Starbase 
returns the display list number, a label number (if any), and the offset from the label. With 
this information, R2 can decide which object and polygon have been selected. 

Translations and rotations result in changes to the transformation matrix: ’amat’. A 
matrix representing the appropriate translation or rotation is made and then multiplied by 
’amat’. The result is put back into ’amat’. The new ’amat’ also replaces the old matrix in the 
display list. 

Attaching an object to another object is a complex procedure. First the object to be 
attached (child object) is selected and then the polygon attachment point is selected. The 
same is done for the base object. R2 then knows the two objects and the polygon faces 
where they are to be attached. The center points of the polygons are computed along with 
the normals to the polygons. Next, the normal direction for each polygon is set to the ’Z’ 
axis. Vectors for the ’X’ and ’Y’ axes must also be constructed for each polygon. Two 
matrices are created that represent the positions and orientations for the point of attach- 
ment. The inverse of the matrix for the base object is multiplied by the matrix for the child 
object. This yields a matrix which describes the transformation of the child object in the 
base object’s coordinate frame. This is the transformation on the child object necessary to 
line up the attach points. This matrix is stored in the child object’s ’ref matrix. Also, the 
child object’s ’amat’ is set to identity. This cancels any rotations or translations on the child 
object and forces the two objects to line up as specified. Rotations and translations can be 
done on a child object but will now be relative to the base object. The base object’s ’kids’ 
array is updated to show that the attaching object is now a child of the base object. The ’ref 



matrix is put in the display list for the child object and a call to the child’s display list is put 
at the end of the parent (base) object’s display list. 

Deleting an object would be a simple procedure were it not for the complexity intro- 
duced by attachments. The simplest method of handling this is deleting all children of an 
object that is deleted. However, this is not desirable. Therefore, when an object is deleted, 
all of its children are unattached and restored to normal status. One problem exists: child 
objects positions are defined by matrices that are relative to the parent object’s position. 
Therefore, the child object’s ’ref matrix is not set back to identity, but is instead multiplied 
by the product of its parent’s ’ref and ’amat’ matrices. This results in the object not moving 
from its current position in the world. One can think of this as a virtual object (invisible 
object) existing where the old parent object existed. This virtual object provides invisible 
support to the child objects, preventing them from collapsing inward. 

Resizing an object is another procedure that would be simple if one did not have to 
deal with attached objects. After an object has been picked to be resized and the new 
parameters have been entered, a completely new object is created. If it is a child object, 
then its parent is looked for. The information regarding attachment points is stored in the 
parent. The polygons are the same as before except that the dimensions are different. New 
attachment points are calculated based on the new coordinates of the new object and the 
’ref matrix is calculated. The ’kids’ array of the parent is modified to point to the new 
object and the old object is removed. If the resized object is itself a parent then the old 
object’s ’kids’ array is copied to the new object and all of the ’ref matrices of the child 
objects are recalculated. Also, all references in display lists to the old object are changed to 
the new object and calls to any children are placed in the new display list. The old object’s 
display list is removed and the memory allocated to the object is freed. 

When the link (or other structure) is complete, it is saved in a form able to be read by 
R2. R2 can not take ROBOSIM code and create editor structures from that. Therefore, 
one must save any files that might possibly be edited again. After all the links of a robot 
have been edited, ROBOSIM code can be generated. Currently, the editor is set up in such 
a way that after generating the ROBOSIM code, ROBOSIM is automatically called and the 
appropriate filename passed to it. ROBOSIM then generates the link files for the robot. If 
the user is on a terminal capable of using MD, then MD is automatically executed with the 
robot name passed to it. In this manner, it is much quicker and more flexible to use the 
editor, since the user does not have to exit the editor, run ROBOSIM, and then run MD. 

R2 generates ROBOSIM code in a fairly straightforward, though not necessarily 


intuitive (especially when looking at the ROBOSIM code), way. The method used resulted 
from the difficulties involved in creating more "readable" ROBOSIM code. One method 
would have required a breadth-first traversal of the editor’s hierarchical structures starting 
at the deepest level of the tree (the thickest part). Another method would have required 
more registers than ROBOSIM has if there were more than four child objects to any object. 
The method used can be thought of as resolving the hierarchical structure dependencies 
into a simple list. Remember that the position and orientation of an object affects all of its 
children objects by the fact that the children’s position and orientation are described in the 
coordinate frame of the parent. All that has to be done is multiply all of the transforma- 
tions down the tree and get one absolute transformation for each object. Then, the first 
object is created, moved and rotated, and then stored in register B. Each additional object 
is handled the same way except that register B is added to it and the result stored in 
register B. Once all the objects in a link have been processed, a "STORE-LINK" command 
is added. Each link is processed the same way until no more links are left. 

Pictures 1 through 5 show two links of a robot being built. First, a cylinder is created 
and moved to one side. Then, a custom object is created. Next, the custom object is at- 
tached to the cylinder. The final steps for this link are a fixed joint attached to the base of 
the cylinder and a revolute joint attached to the custom object. Then, the link is saved. 
Another link, a simple box, is created. The input and output joints are made and attached 
to the link. Then, that link is saved. After these links have been saved, ROBOSIM code is 
generated for them and passed to ROBOSIM. The output from ROBOSIM can be seen, 
also. Now, the files describing these two links have been created and they can be looked at 
with MD. The ROBOSIM code generated for the base link (LOC link) is listed in Table 1. 
The structure of the link file generated by ROBOSIM is shown in Table 2. 



LOOK-FROM X=-100., Y=100., Z=45. 
LOOK-AT X=0., Y=0., Z=8. 

CLEAR 

• STORE B 
R-JOINT-I+1 

ROTATE X = -45.000 
ROTATE Z= 90.000 

TRANSLATE X=-5.000, Y= -30.000, Z=55.000 
ADD B 

• STORE B 
CLEAR 

MOVE X= -10.000, Y= -10.000, Z= 0.000 
DRAW X= -10.000, Y = 10.000, Z=0.000 
DRAW X= 15.000, Y= 10.000, Z= 0.000 
4 DRAW X= 25.000, Y= 0.000, Z= 0.000 

DRAW X= 15.000, Y= -10.000, Z= 0.000 
DRAW X= -10.000, Y= -10.000, Z= 0.000 
EXTRUDE-SURFACE Z= 10.000 
ROTATE X= -90.000 
ROTATE Y= -90.000 

TRANSLATE X= 0.000, Y= -30.000, Z= 35.000 

• ADD B 
STORE B 
CLEAR 


F-JOINT-I 

TRANSLATE X=0.000, Y= -30.000, Z= -25.000 

• ADD B 
STORE B 
CLEAR 

CYLINDER R = 10.000, H = 50.000 
TRANSLATE X=0.000, Y=-30.000, Z= 0.000 
m ADD B 

W STORE B 

CLEAR 

LOAD B 

STORE-LINK C.LOC 
VIEW 

• END 


Table 1. ROBOSIM code generated by R2 
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Pictures 2a-b: Cylinder and custom object before and after attachment 
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Pictures 4a-b: Base link being saved and compiled by ROBOSIM 
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Pictures 5a-b: First link being saved and compiled by ROBOSIM 








Row 

Col 1 

Col 2 

Col 3 

Col 4 


• 

1 

| THETA 

| DZ 

| DA 

ALPHA 

j 


2 

| JA1 

| JA2 

| JTYPE 1 

JTYPE2 

I 


3 

1 

AINERT 

(4X4) 


I 


7 

1 

AJNT-I 

(4X4) 


i 

• 

11 

1 

AJNT-I+1 (4X4) 


i 


15 

1 

AMAT 

(4X4) 


I 


19 

| NVEC 

| UNUSED 

| UNUSED 

UNUSED 

I 

• 

20 

1 XI 

1 xi 

1 zi 

Dl 

i 


21 

| X2 

| Y2 

| Z2 

| D2 

7 


NVEC+19 | XNVEC | YNVEC | ZNVEC | DNVEC 


Variable Definitions: 


THETA 

DZ 

DA 

ALPHA 

JA1 , JA2 

JTYPE- 1,1+1 

AINERT 

AJNT-I , 1+1 

AMAT 

NVEC 

Xi,Yi,Zi 

Di 


Denavit-Hartenberg parameter 
Denavit-Hartenberg parameter 
Denavit-Hartenberg parameter 
Denavit-Hartenberg parameter 
joint defined flag 

joint type -> Revolute, Prismatic, Fixed 

generalized link inertia 

transforms of input and output frames 

link's A-matrix 

number of vectors in list 

x,y, and z component of vector 

move or draw vector 


Table 2. Structure of Link File Created by ROBOSIM 


27 



2.4 Simulatio n Library and Environment 


The simulation library and environment provides methods to access the data struc- 
tures created by ROBOSIM. The robots and other objects are specified and loaded into 
memory. These structures remain resident in memory while the simulation is running. The 
library provides an interface to these structures so that the user does not have to under- 
stand what is happening at that level. The library provides higher level facilities much like 
an actual robot programming language. 

The simulation package allows one to use the robots that have been designed. The 
package consists of a library of C functions that operate on the files created by ROBOSIM. 
Although this package is far from complete, it allows simple simulations to be run. Also, it 
provides a framework in which to test the major components for the simulator: collision 
detection and dynamics. Having the simulator be a library of C routines allows more 
flexible methods for running simulations. Very specific and efficient simulations can be 
written in C and which call the simulation functions directly. However, even at this level, 
much of the internal data structures is hidden from the user. This level of programming 
roughly corresponds to programming a robot in its programming language. For instance, 
one can tell a robot to move along a straight line or move a particular joint. A complete 
reference of simulation functions available can be found in Appendix 2.1. Using these same 
routines a very flexible, user-friendly interface can be built up, allowing an interactive way 
to do simulations that are not too complicated, or that do not require great speed. 

ROBOSIM provides most of the information required by the simulator by way of the 
files it creates. However, some information is not directly provided, but it can be deter- 
mined from what is there. This involves the information required by the collision detection 
algorithm. ROBOSIM provides the Denavit-Hartenburg parameters, the A matrix, the 
pseudo-inertia matrix, and a list of points which describe the physical structure of the robot. 
The internal data structure also includes areas that are not currently used, but will be at a 
later time. These include minimum and maximum joint angles, velocities, and accelera- 
tions. The structure also includes information related to Starbase graphics. The actual C 
structure declarations used can be found in Appendix 2.1. The simulation package acts as 
intermediary between the user and the internal representation. 

The simulation program that the user writes can turn on collision detection, request 
solutions to inverse kinematics problems, and display results graphically. The user can use 
the general numerical Jacobian method for inverse kinematics or provide an exact solution 



for his robot. The user simply passes the address of the function to the simulator, and the 
simulator will then use that function when solving inverse kinematics for that robot. A 
proposed extension to the simulator will allow the recognition of the twenty-four possible 
robot configurations for which exact solutions exist. The exact solutions to these configura- 
tions would then be used instead of a numerical method, freeing the user from having to 
solve and code it himself. A good use of the simulation system can be found in a later case 
study section. This case study uses most of the features of the simulation system, as well as 
R2. 


The simulation library’s commands correspond to real robot programming commands 
found in many robot languages. Interfaces to many different robot languagues are planned. 
This will allow actual robots to be simulated, and then have a verified program downloaded 
to the robot. Additionally, the simulation could be run in parallel with the robot, with a 
planner or some other type of higher-level process sending the same commands to the 
simulation as well as to the actual robot. This can be used for verification, or even more 
importantly as part of a feedback loop to the planner. This will allow the planner to receive 
information from the simulation that it can not get from the actual robot. For instance, the 
simulation could provide forces and torques if the robot does not have sensors for that. 
Also, the planner could check out a plan of action on the simulation before actually driving 
the robot. This would let the simulation check for collisions or other dangers without 
risking the real robot. 


2.5 Inverse Kinematics 


The current default method for solving the inverse kinematics problem is the 
Newton-Raphson method. This method is an iterative method which uses the Jacobian of a 
robot. It is limited to six degree of freedom arms and has many other problems. The 
jacobian is a six by six matrix that relates differential changes in joint angles to differential 
changes in world coordinate space. In other words, if you take the vector of joint velocities 
and premultiply it by the jacobian the result will be the velocities of the end effector in 
coordinate space. Now, if you invert the jacobian matrix, then you have a matrix that 
relates differential changes in coordinate space to differential changes in joint angles. Now, 
if the robot end effector is at a certain place and you want to know what joint variables 
would put it there then do the following procedure. 

First, record the current joint angles. Then, compute the jacobian and invert it. Now, 


subtrace the current position of the robot in coordinate space from the desired location in 
coordinate space. Multiply the inverse jacobian by this difference. This yields a set of 
differences in the joint angles. Add this set of differences to the joint angles. Compute the 
new position of the end effector. Iterate this procedure until the error is acceptable low. 

There are many problems with this procedure. First, due to singularities in the 
jacobian, the method often does not converge. Second, when it does converge, you get only 
one possible solution and there is no way to get the others. Third, it is very slow. However, 
there are some robot configurations in which there is no exact solution, and therefore this 
is the only general way. 

The implementation used does not yield very good results. However, it is faster than 
that used in the original ROBOSIM. This probably results from the use of LU decompisi- 
tion instead of actually computing the inverse of the jacobian. For example, if you are 
trying to solve the matrix equation ( Y = JX ) for X, one way would be to invert J and 
premultiply both sides by that. However, there is a faster way to solve this. J can be ex- 
pressed as the product of two matrices, an upper diagonal matrix and a lower diagonal 
matrix. With J in this form, X can be solved by back substitution. 

The best way to solve the inverse kinematics problem is to provide the exact solution. 
Although this is usually difficult, there are only 24 distinct configurations. This means that 
if a robot has an exact solution, its inverse solution can be expressed by one set of equa- 
tions out of a possible 24. Currently, only one set of equations is implemented: the one 
corresponding to the PUMA 560. However, it is in a general form, in which six parameters 
( the lengths of the links) can vary. This method also allows one to get all possible solutions 
to the problem. In this way, additional constraints can be checked for, such as limitations of 
joints and checking different solutions to find one that does not collide with itself or other 
objects. This method is also faster by two orders of magnitude. Also, most commercially 
available robots have configurations that have exact solutions. It is not possible to run a 
simulation in real time using the numerical method, at least not without a floating point 
accelerator. 

The only other method involves solving for five of the six joint angles analytically and 
using the Newton-Raphson method on one joint This method will work on some configura- 
tions that do not have exact solutions. The usefulness of the method is better, yielding more 
solutions than the full Newton-Raphson method. In addition, the numerical part of the 
algorithm is not as sensitive to singularities, since it involves only one equation. This 
algorithm is not currently provided in the simulation, but the user could provide his own. 
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2,6 Collision Detection 


Collision detection is very important in simulation of robots. One usually wants to 
know if the robot has collided with its environment or with itself. The following discussion 
does not delve into the theory behind the methods used, nor does it give an overview of 
collision detection. For a complete discussion of collision detection methods see Walter’s 
dissertation from Cornell. The collision detection algorithm implemented here is very 
similar to the POCODA (Polygon Collision Detection Algorithm) algorithm given by 
Walter. The implementation used is given with special emphasis on those extensions to 
POCODA. 

The algorithm used can be broken down into several subalgorithms. These will be 
discussed from lowest level to highest level. The assumptions used here is that all objects 
are defined by convex planar polygons. The problems involved in collision detection are as 
follows. Given a polygon and a point in the plane of the polygon determine whether that 
point is inside of the polygon. Given a polygon and a line segment determine whether the 
line segment crosses the plane of the polygon. Given two polygons determine whether they 
intersect. Given two objects determine whether they intersect. Given two bounding 
volumes around two objects determine whether they overlap. 

The one equation to keep in mind throughout this discussion is the plane normal 
form of the plane equation. 


phi(P) = N . P + nd 


(Eq. 1) 


Where: 

N: normal to the plane 
P: point 

nd: distance from plane to origin 

The plane described by this equation is the set of points P such that phi(P) is zero. 
Also, given N, nd, and a point P, the residue (phi) is zero if P is in the plane, positive if P is 
above the plane, and negative if P is below the plane. 

The point-in-polygon problem is the most time-consuming operation. The method 



used to solve this problem is the reason why the polygons must be convex. The algorithm is 
to follow the polygon’s edges around the polygon checking to see which side of each edge 
the point is on. If the point is to the same side of each edge then that point is inside of the 
polygon. This is checked by substituting the point into each edge’s penalty function. The 
penalty function is a plane equation such that the edge lies in the plane and the plane is 
perpendicular to the plane of the polygon. The penalty function is calculated once for each 
edge and stored in the internal structure. See Appendix 2.1. for the C simulation structure. 


pen(P) = M . P + md 


(Eq. 2) 


(NxE) 

M = (Eq. 3) 

l|E|| 


md = - M . Pie 


(Eq. 4) 


M is the normal vector to the penalty plane; it is the cross product of the normal to 
the polygon plane and the directed edge normalized with respect to the directed edge, md 
is the distance of the penalty plane from the origin. This penalty function can now be used 
to determine which side of an edge a point is on. 

The algorithm for determining if a line segment crosses the plane of a polygon should 
be obvious from the above discussion. The two endpoints of the line segment are both 
substituted into the equation of the plane in which the polygon lies. If the residues of the 
two points are the same sign then both points lie on one side of the plane. Therefore, the 
line segment did not cross the plane. If, however, the residues have different signs, then the 
point at which the line segment crosses the plane must be determined so as to use it in the 
point-in-polygon algorithm. Given two points PI and P2 which are the endpoints of a line 
segment and phi(Pl) and phi(P2) which are the residues of PI and P2 in the polygon plane, 
then the point along the line segment that intersects the polygon plane is Pc, 


(P2 - PI) phi(Pl) 

Pc = PI + 

phi(Pl) - phi(P2) 


(Eq. 5) 



Each object in a simulation is composed of polygons, but due to speed and efficiency 
requirements the above tests would be prohibitive. Therefore, some simpler tests are 
required which can quickly eliminate some objects from the more exhaustive tests. The 
method used is to perform tests on bounding boxes of the objects. A bounding box is 
described by a point and a vector. The point is the center of the box, and the vector is the 
half-diagonal vector of the box (i.e. it points from the center of the box to a comer). These 
values are determined by first determining the maximum and minimum values of the object 
along the x, y, and z axes. The center is calculated by averaging the maximum and mini- 
mum values along each axis. The half-diagonal vector is calculated by taking half of the 
difference between the maximum and minimum along each axis. For instance, along the X 
axis: 


Xmax + Xmin 

Cx (Eq. 6) 

2 

Xmax - Xmin 

Dx = (Eq. 7) 

2 

Now, two bounding boxes overlap if the distances between the centers along every 
axis is less than the sum of the half-diagonal components along the corresponding axes. 
However, the two bounding boxes must be defined in the same coordinate frame. Typically, 
each object is defined in its own coordinate frame and has a transformation matrix describ- 
ing the position and orientation of the object in the world coordinate frame. Therefore, a 
method is needed to transform a bounding box from one frame to the other. Given two 
bounding boxes, B1 = {C1,D1} and B2 = {C2,D2}, and two transformations T1 and T2 
which are 4x4 matrices describing position and orientation of boxes B1 and B2, respec- 
tively, let Cl, 2 and Dl,2 be the center and half-diagonal vector of B1 in coordinate frame 2. 


Cl, 2 = [C1][T1]([T2]-1) 


(Eq. 8) 


Dl,2 = D1 @ [T1]([T2]-1) 


(Eq. 9) 



where @ is the dilation product, an operation between two matrices which can be 
expressed as the product of two matrices whose elements have all been changed to their 
absolute values. 

In order to test for bounding box overlap given two boxes, one first has to express B1 
in coordinate frame 2 and check for an overlap. Then convert B2 to coordinate frame 1 and 
check for an overlap. Only if both checks indicate an overlap is there one. If an overlap is 
indicated then further checks have to be made to determine if there is a collision. 

Once a possible collision is indicated by overlap of bounding boxes, more exhaustive 
tests have to be performed. First, all points in one object must be transformed to the other 
object’s coordinate frame. This can be done using Eq. 8 above where Cl is a point in object 

1. Once this is done, a first approach would be to check every edge in each object against 
every polygon in the other object. However, there are some ways to reduce the number of 
edges which must be checked. First, each edge in object 1 is checked against the bounding 
box of object 2. Only if the edge falls within the bounding box could it intersect the object. 
Each edge that could intersect a polygon is saved in the reduced edge array. Now, each 
edge in the reduced edge array is checked against the polygons in object 2. However, each 
polygon from object 2 is first checked to see if the plane it lies in could intersect the bound- 
ing box of object 1. If it does not, there is no need to check edges against it. Finally, each 
possible edge is checked against each possible polygon, using the methods described above, 
to determine if a collision exists. If not, then all points of object 2 are transformed to the 
frame of object 1 and the procedure repeated. The following summary is from Walter’s 
thesis. 

1. Compare the bounding boxes of each object. 

(a) If the bounding boxes overlap then the likelihood of a collision is high and 
further checks are required, and the procedure continued. 

(b) Otherwise the two objects cannot possibly collide. They may be declared 
collison-free, and the procedure is exited. 

2. The objects are transformed to a common reference frame by tranforming the 
points of j into the reference frame of k. The new object is referred to as (j,k). 

3. Edges in (j,k) are compared with the bounding box of k. 



(a) If an edge intersects the bounding box it is retained for further tests by insert- 
ing it into the reduced edge array. 

(b) Otherwise the edge is excluded from further tests. 

4. Check each polygon in k. 

(a) Check whether the polygon plane intersects with the bounding box of (j,k). It 
means comparing the polygon against all the reduced edges in (j,k). 

A. If the polygon intersects with an edge then a collision has occurred, and 
the procedure is exited with a collision condition. 

B. Otherwise, continue until all edges are considered. 

(b) Otherwise, the polygon cannot possibly be a source of collision, and is ex- 
cluded from further tests between the two objects. 

5. Evaluate progress. 

(a) If this is the first time to this step then, interchange the roles of j and k and 
repeat all steps after (2), since a collision is still possible, although undetected 
this far. 

(b) Otherwise, the two objects do not collide. They may be declared collision-free, 
and the procedure exited. 

This algorithm is the one used in the simulation library and environment with one 
difference. At step 3, Walter checks every edge in (j,k) against the bounding box of k. A 
much simpler first check is to check the plane of the polygon that contains the edge against 
the bounding box first. If that polygon does not intersect the box, then all the edges of that 
polygon are excluded. This is a much faster check than checking an edge against a box. This 
is similar to what is done for the k object in step 4. 

ROBOSIM does not directly generate all the information required for collision 
detection. However, it can be calculated from what is provided, namely the vector list. The 
vector list is a list of points that define the polygons of the object. This vector list is split 
into separate polygons as it is read from a file. Then the normal and normal distance for 
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each polygon is calculated and stored. Next, the penalty function for each edge is calculated 
and stored. As the vector list is read in, the maximum and minimum x, y, and z values are 
saved and used to calculate the bounding box. The internal data structure now contains all 
of the information necessary for collision detection. 

The use of this algorithm requires some special considerations when used with robots. 
The technique used employs a bounding box around each object in the environment, a 
bounding box around each link of each robot, and a bounding box around each robot. The 
bounding boxes around each object and each link are computed at load time, but the 
bounding boxes around robots must be computed as needed. This is because the bounding 
boxes around robots change as the joint angles in the robots change. Whenever a collision 
is checked for, bounding boxes are created around the robots. They are calculated by using 
the bounding boxes around the links. The minimum and maximum extents along the x, y, 
and z axes of the bounding boxes around the links are computed. Then a bounding box 
around all these bounding boxes is computed from the minimum and maximum extents. 
The purpose for bounding boxes around robots is that if there is more than one robot, even 
bounding box checks become expensive. If there are two robots, each with nine links (6 
movable and 3 fixed), 81 bounding box checks would be required every time. And if there 
were three robots, 729 bounding box checks would be required. With three robots, and 
therefore three bounding boxes, only three bounding box checks are required. If there is a 
collision between two bounding boxes, only the two robots need be checked. 

Previously, there was a transformation matrix associated with each link that described 
the coordinate frame of that link with the previous link. This is not adequate for collision 
detection, however. This matrix can be obtained by multiplying all of the matrices of the 
previous links together, yielding a transformation of the current link in the world coor- 
dinate frame. It is much simpler, and faster, to calculate this matrix for each link whenever 
joint variables are changed in the robot rather than waiting until needed by collision 
detection. This is especially true since collision detection checks are made from the end 
effector inward, as a collision is more likely with the end effector. Whenever a joint vari- 
able in a robot is changed by a library function, the transformation matrix of the link in the 
previous link’s frame as well as the world frame is calculated and stored in the link’s struc- 
ture. Then, it is used by the collision detection algorithm as needed. 

Another problem that requires special treatment is collisions involving the robot with 
itself. This is especially difficult when one considers that the design of the robot may 
include overlap of adjacent links. If this is the case, then if links of the robot are checked 
with other links of the same robot, then collisions might be seen that aren’t really valid. 


Therefore, collisions are not checked for against adjacent links. The simulator has internal 
provisions for joint constraints. Therefore, any possible collision could be provided for by 
limiting the joint angles. However, given legal joint values, it is possible for non-adjacent 
links to collide. Therefore, collision detection of the robot with itself must be made. Given 
a nine link robot, 28 bounding box checks must be made to ensure no collisions with itself. 
However, this self-collision detection may be controlled separately (i.e. it can be turned on 
and off independently of the other collision detection), since the user may not require these 
tests. 


Since many objects and robots may be loaded before they are actually used in the 
simulation, the collision detection uses the list that is created by the USE command. The 
USE command adds its argument to a linked list of objects, and inserts a call to it in the 
display list. Therefore, it will be displayed when the display list is traversed. Also, the 
collision detection uses the linked list of objects to check for collisions. If an object is not in 
use, the collision detection does not waste time checking it. 

Once collision detection is turned on, checks for collisions are made any time the 
library functions are used to move a robot. If there is a collision then a collision structure is 
filled out. This structure returns pointers to the objects and link numbers if the objects are 
robots. The library functions pertaining to collision detection are included in Appendix 2.2. 

The collision detection algorithm has only two weak points. It does not handle con- 
cave polygons, and it will not signal a collision if one object is completely inside of another. 
The stipulation concerning concave polygons is not serious. ROBOSIM does not generate 
concave polygons unless they are the result of an custom object. Although R2 does not 
check for concave polygons, this feature could be implemented. In fact, algorithms exist to 
split concave polygons into convex polygons. Either of these features could be implemented 
fairly simply. The problem of not detecting a collision if one object is completely inside 
another derives from the fact the algorithm used is a polygonal collision detection algo- 
rithm and not a solid object one. However, assuming two objects start off outside of each 
other and movements are sufficiently small, then this should not prove to be a problem. 
This condition also prevents the ability of one object to pass through another (i.e. a move- 
ment is large enough that two objects do not overlap at any point). This algorithm does not 
detect collisions in the volume swept by an object moving between positions with another 
object, but rather only overlap of the objects at the starting and ending positions. But, if the 
distance between the positions is smaller than the smallest object, then there should be no 
problems. 



The collision detection has been implemented very effectively. The low level collision 
routines require transforming points in one coordinate frame to the other. This requires 
multiplying all points by a transformation matrix. The Starbase graphics package provides 
routines to do this, as well as to multiply 4x4 matrices together. When there is a graphics 
accelerator in the system, Starbase uses it to do the calculations. This allows matrix multi- 
plication as well as transformation of points to be done in hardware, which is much faster 
than in software. 


2.7 Surgical Positioner 


Everything described up to this point has been tested, and is in use. R2 and the 
simulation package are being used presently to aid in designing a kinematic surgical 
positioner. Its application would be specifically for brain surgery. The idea behind it is this: 
the robot would not be capable of motion on its own. It would be attached to a surgical 
collar, and after calibration would be positioned by the surgeon, with joint encoders sen- 
ding the values of the joint angles to a computer. The computer would show the position of 
the robot superimposed on a CAT scan. In this way, a surgeon can quickly determine points 
of entry. Currently, this is accomplished by precomputing where the points would be and 
then determining them using the collar as a reference. Having a way to immediately see 
what the positions are would prove to be much more flexible. Additionally, a hollow tube 
could be attached to the end effector. With this, the robot could maintain a particular 
orientation while the surgeon takes a biopsy. 

Current research is to determine whether a robot of sufficient accuracy can be built. 
ROBOSIM provides an excellent test bed to perform this development. R2 has been used 
to design the arm and specify the dimensions of the links. A basic configuration similar to 
that of the PUMA 560 has been used. Therefore, an exact inverse kinematics solution exists 
and is used. The simulation library and environment is used to test the arm. The tests 
include ability to reach all the required points on the head (without passing through it). 
The inverse kinematics equations generate eight different solutions. These solutions are 
checked using the collision detection algorithm to ensure that there exists at least one 
which will reach the desired position without touching the head. 

An additional requirement is that the positional accuracy of this robot be small. 
However, the size and cost are also important factors, so the smallest joint encoders would 
be desirable. The relation of world positional accuracy to joint accuracy is fairly easy to 



determine. Given a joint encoder of a certain number of bits, the accuracy is the range 
divided by two to the number of bits. This gives an angular measure of the amount a joint 
encoder could be off. This is used with the jacobian to determine the maximum positional 
error. The jacobian relates differential changes in joint angles to differential changes in 
world coordinates. The error in position caused by each joint is first determined. Then, the 
sum of the errors is computed. This gives the maximum amount that the positioner could 
be off. (It assumes each joint is off in the direction to give maximum error.) 


Once a robot is generated, the simulation can run without the user. All data is saved 
in a file for later analysis. The simulation can run without displaying any graphics, or the 
user can watch it as the robot is put through its paces. The part of the simulation written by 
the user is shown below. It is not a general type of simulation that would be applicable to a 
wide variety of problems. However, it is sufficiently general in that it encapsulates the 
requirements of the project, but it does so without being limiting. For instance, the require- 
ments are that it reach certain points on a head (cylinder) without any part of the robot 
touching the head. The user cycles through the points that are required, and the simulation 
sends back information concerning whether the robot specified can reach the points 
without colliding with the head. 

#include "sim.h" 

#include <math.h> 

#include <stdio.h> 

#defme TRUE 1 
#define FALSE 0 

int i; 

FILE *fopen(), *fpout; 
int puma_inv(); 

sim() 

{ 

ROBOT rl=0; 

OBJ ol=0; 

JOINT array; 
float J[6][6]; 
float angle; 

extern COLLISION S CO; 
char ‘filename = "testout"; 
float m[4][4]; 

float CONV = M PI/180.; 

/* 

* PRE does transformation along world axes 

7 

rl = GET_ROBOT("/users/robosim/source/manipulators/jo/models/T"); 
PRETRANSLATE(rl,25.,0.,0.); 



USE(rl); 

ol = GET_OBJ("/users/robosim/source/manipulators/jo/models/HEAD.OBJ"); 
PRETRANSLATE(ol,0.,0.,0.); 

USE(ol); 

CSWITCH(TRUE); 

SET_INV(rl,puma inv); 

/* " 

* cover head in increments of 1cm over the length and 5 degrees 

* from 75 to 295 output results to file testout 

7 

fpout = fopen(filename,"w"); 
setJoint_error(); 

for (i= -20; i<21; i+ +) { 

for (angle =0.; angle <105.*CONV; angle + =5.*CONV) { 
get_location(angle,m); 
fprintf(fpout,"\n"); 

fprintf(fpout,"%5.2f %5.2f %5.2f %5.2f %5.2f %5.2f\n\ 
lo[0],lo[l],lo[2],lo[3],lo[4],lo{5]); 
if(KINV(rl,m, array, TRUE)) { 

JACOB(array r J); 

error(J); 

if(!MOVEJI(rl, array, 1)) { 

printf("collision, y = %d, angle = %f\n",i, angle); 
printf("link %d\n",S CO.L1); 

} 

} 

eke { 

fprintf(fpout,"pomt did not converge\n"); 
printf("point did not converge\n"); 

} 

} 

} 

fclose(fpout); 


get_location(j,m) 
float j; 

float m[4][4]; 

( 

m[2][0] = -cosO); 

m[2][2] = -sin(j); 

m[2][l] = 0.; 

m[0][0] = m[0][2] = 0.; 

m[0][l] = 1.; 

cross(m[2],m(0],m[l],l.); 

m[3][0] = (float) -20.1* m[2][0]; 

m[3][2] = (float) -20.1* m[2][2]; 

m[3][l] = (float) i; 

m[0][3] = m[l][3] = m[2][3] = 0.; 

m(3][3] * 1.; 

return(l); 


} 



float NUMBITSQ = { 12.0, 12.0, 12.0, 12.0, 12.0, 12.0}; 
float single joint_error[6J; 

setjoint_error() 

{ 

int i; 

for (i=0; i<6; i+ +) { 

single joint_error[i] = 2.*M_PI/pow(2.0,NUM BITS[i]); 

} 


error(m) 
float m[6][6]; 

{ 

int ij; 
float err[6]; 

for(i=0; i<6; i+ +) { 
err[i] = 0.; 
for(j=0;j<6y++) { 

err[i] + = (float)fabs((double)m[i][j] * single joint_error[j]); 

} 

} 

fprintf(fpout,"dX = %f dY = %f dZ = %P,err[0],err[l],err[2]); 
fprintf(fpout," rX = %f rY = %f rZ = %f\n*,err[3],err[4],err[5]); 
fprintf(fpout, "distance error %f\n", 

(float) sqrt((double) err[0]*err[0]+err[l]*err[l]+err[2]*err[2])); 


Currently, various configurations with twelve bit joint encoders are being investigated. 
It appears that twelve bit encoders will provide the necessary accuracy. The use of R2 and 
the ability to resize objects provide a simple means to quickly create a new configuration. 
The generalness of the simulation library allows the same simulation to be used with no 
modification. A detailed description of the simulation library commands is provided in 
Appendix 2.2. 


APPENDIX 2.1 Structure Declarations for the Simulation Library 


This Appendix contains all structure declarations used throughout the code of the 
Simulation Library. The declarations are given using the conventions of the C program- 
ming language, since the Simulation Library itself was coded in C. 


typedef float (*S_VECTOR)[3]; 


struct s_penalty { 
float pen_norm[3]; 
float pen dist; 

}; 


typedef struct s_poly { 
float norm[3]; 
float nd; 
int vec_ptr; 
int num_vectors; 
struct s_penalty *pen; 

} *S_POLY; 

typedef struct s_link{ 
int num vectors; 

S_ VECTOR list_ptr; 
float *md; 
int display_list; 

float bbc[4]; /’bounding box center*/ 

float bbd[4]; /’bounding box half-diagonal*/ 

float INERT[4][4]; 

float CURR[4][4]; 

int num_poly 

S_POLY poly, 

float theta,dz,da,alpha; 

int jtypel jtype2; 

float JNT1[4][4]; 

float JNT2(4][4]; 

float AMAT[4][4]; 

float TRANS[4][4]; 

float curr_var; 

float min_var; 

float max var; 

} ’S LINK; - 

struct s_robot{ 
int num links; 

S_LINK _ link[18]; 
float Pre[4][4]; 
float Post[4][4]; 
float DH[18][5]; 
int displayjist; 

float POS[4][4]; /’matrix describing position of robot in environ*/ 
int (*INV KJN)(); /* pointer to function that solves inverse kin */ 

}; 


struct s_env{ 
int num_vectors; 

S VECTOR list_ptr; 
float *md; 
int display_list; 

float bbc[4j; /’bounding box center*/ 
float bbd[4]; /’bounding box half-diagonal*/ 


float INERT[4][4]; 
float POS[4][4]; 
int num_poly, 
SPOLY poly, 

}; 


/* this is a copy of s_env, however it is also the generic type 
/* of which link and obj can be cast into */ 
struct s_gen{ 
int num vectors; 


SVECTOR list_ptr; 
float *md; 
int displaylist; 

float bbc[4]; /’bounding box center*/ 
float bbd[4]; /’bounding box half-diagonal*/ 
float INERT[4][4]; 
float POS[4][4]; 
int num_poly, 

S POLY poly, 

}; " 
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struct s_obj { 
int numvectors; 

S_VECTOR list_ptr; 
float *md; 
int display_list; 

float bbc{4]; /’bounding box center*/ 

float bbd[4]; /’bounding box half-diagonal*/ 

float INERT[4][4]; 

float POS[4][4]; 

int num_poly 

S_POLY poly, 

float DIFF(4][4]; 

}; 


typedef struct s_any { 

int type; /* 0=robot l=env2=obj */ 
int in_use; 
union { 

struct s robot *r; 
struct s_env *e; 
struct s_obj *o; 

} o*>j; ~ 

} *S_ROBOT,*S_ENV,*S_OBJ; 

typedef struct s_list { 
struct s_any ’item; 
struct s list ’next; 

} S_LIST;~ 

typedef struct s_collision{ 
struct s_any *S1; 
int LI; 



struct s_any *S2; 
int L2; 

} SCOLLISION; 

typedef float S_POINT[3]; /* X Y Z */ 

typedef float S_ORIENTATION[3]; /* ROLL PITCH YAW */ 

typedef float S_LOCATION[6]; /* X Y Z ROLL PITCH YAW */ 

S ROBOT S GET ROBOT(); 

S'ENVS GET_ENV0; 

S~OBJ S_GET_OBJ(); 
typedef float SjjOINT[18]; 

typedef float matrix3d[4][4]; 

struct matrix_struct {matrix3d msxform; }; 

# define S_REPLACE_MATRIX_3D(dest,src) \ 

‘(struct matrix_struct *)(dest) = ‘(struct matrix_struct *)(src) 


APPENDIX 2.2 Simulation Library Functions 


This Appendix contains the interface declarations to the functions of the Simulation 
Library. The declarations are given using the conventions of the C programming language, 
since the Simulation Library itself was coded in C. 


S_GET_ROBOT (filename) 
char ‘filename; 

S_GET_ENV (filename) 
char ‘filename; 

S_GET_OBJ (filename) 
char ‘filename; 

S PRETRANSLATE (opc,y,z) 
~S_ANY o; 
float x,y,z; 

S POSTTRANSLATE (o,x,y,z) 
~S_ANY o; 
float x,y,z; 

S PREROTATE (opc,y,z) 

~S ANY o; 



S POSTROTATE (o,x,y,z) 
"S_ANY o; 
float x,y,z; 

S MOVEJ (rjoints) 
“SROBOT r; 

SJOINT joints; 

SCLEARJOINT (joints) 

S JOINT joints; 

S MOVEJI (rjoints, steps) 
"S_ROBOT r; 

S~JOINT joints; 
int steps; 

S USE (o) 

"S_ANY o; 

S DONTUSE (o) 

"S_ANY o; 

S CHECK (o) 

“S_ANY o; 

S CHECK ROBOT (r) 
"S_ANY o; 

S_C_S WITCH (x) 
int x; 

S_COLLIDE 0 

S SET INV (r, rnv func_ptr) 
"SROBOT r; 
int (*inv_func_ptr)(); 

S KINV(r, dim, joints, reset) 
"SROBOT r; 
float dlm[4][4]; 

S JOINT joints; 
int reset; 

S_JACOB(joints, jac) 

"S JOINT joints; 
float jac[6][6]; 

s_translate (mat^t,y,z) 
float mat[4][4],x,y,z; 


s_rotate (matpc,y,z) 


float mat[4][4],x,y,z; 

s_rotatez (mat,z) 
float mat[4][4],z; 

s transpose (mat) 
'float mat[4][4]; 

s_invert (mat) 
float mat[4][4]; 



3. INTELLIGENT GRAPHICS MODELING ENVIRONMENT 


The ROBOSIM package, together with the enhancements described in the previous 
chapter, provides a powerful graphic tool for designing and simulating geometrical objects 
(including robots, of course) using an engineering workstation. But the real power of this 
approach can be utilized only by integrating the services of a graphic modeling toolkit with 
knowledge-based techniques. This chapter describes the ongoing research efforts to create 
such an integrated modeling environment. First a critical review of the current graphical 
modeling techniques is given, followed by the system design and implementational con- 
siderations of an enhanced graphics modeling and simulation package. Finally the 
knowledge-based techniques developed at Vanderbilt for the integration of large-scale 
engineering systems (instrumentation, control, robotics, simulation, etc..) are summarized. 


3.1 Critique of the Current Graphics Modeling Technique 


The extensions to the ROBOSIM package described in the previous chapter greatly 
enhanced its capabilities in modeling different geometrical objects and systems. But we 
think that a graphics modeling environment should provide some additional features in 
order to fully utilize the potential of knowledge-based techniques in the graphic simulation 
of geometric systems. These additional features are summarized below: 

Need for separate representation of objects: Currently the ROBOSIM modeling environment 
does not support the separate representation of different graphic objects in its 
workspace. The display lists representing these objects are concatenated together 
every time a new object is added to the system. This makes the modification of 
complex objects very difficult, because the whole ROBOSIM command sequence 
creating the complex object must be re-executed whenever one of its parts is 
modified. This is especially a problem during the editing phase, since such operations 
are quite frequently needed here. The solution would be to maintain these objects 
separately -- at least during the editing phase of the modeling. On the other hand, 
concatenating together the parts of a complex solid object would speed up the graphic 
simulation, so the desirable solution is to maintain both representation forms and use 
the appropriate one for each step of the modeling process. 



Need for more graphics objects in the workspace: A large graphics simulation program 

typically contains several independently moving objects. The programming model 
offered by ROBOSIM (graphic registers) limits the number of these objects - i.e. the 
complexity of the systems which can be modeled with it. The desirable solution is to 
allocate the graphic objects dynamically, which does not limit their number. Then 
each of these independent objects could be controlled separately during the simula- 
tion. 

Multiple aspect object representation: Many of the enhancements to the ROBOSIM package 
(collision detection, dynamics, etc..), described in the previous chapter are basically 
"add-on" packages to the original system, with separated data representation schemes. 
The system design could be made much more understandable if a central data base 
would be used, containing every aspect of the models stored in it. 

The next parts of this chapter describe the system design of a planned graphics 
modeling environment which provides the features outlined above. We expect much of the 
already existing ROBOSIM code to be reusable, together with the code for the extended 
features described previously. Unlike the work described in the previous chapter, this is a 
system currently being specified and developed. 


3.2 System Design of the Graphic Simulation Environment 


The system design of the proposed structural modeling environment can be seen in 
Figure 1. It includes two data bases - the Instance Space and the Library Space - with 
several ’active’ components operating on them. The Instance Space contains all data 
structures which describe the current simulated system. It contains data structures describ- 
ing: 

1. the geometrical properties of the entities of the world model, 

2. the part-whole relationships between the entities, 

3. the information necessary for displaying these entities (display lists), 

4. the information necessary for the collision detection algorithm, 



Figure 1. Main functional components of the structural simulation environment 

5. the information necessary for the forward and inverse kinematics simulation of the 
system (either in the form of data necessary for the default iterative methods, or in 
the form of analytical equations if these are available), and 

6. the information necessary for the forward and inverse dynamic simulation of the 
system. 

Basically the Instance Space contains all the information which was necessary to 
operate the models in the enhanced ROBOSIM package described in the previous chapter, 
but in a much better structured form. Unlike in ROBOSIM, where there was a limitation 











on the number of objects which can be handled by the system (fixed number of graphics 
registers), the objects in the Instance Space can be generated dynamically with no preset 
limit on their number or complexity. 

In many cases the objects in the Instance Space are complex structures built of either 
less complex structures or elementary building blocks (like boxes, cylinders, spheres, etc..). 
Frequently there are objects having the same structure but with different parameters of 
their building blocks. The purpose of the Library Space is to store structural declarations of 
these complex objects which can be instantiated with the desired parameters whenever a 
new entity has to be generated in the Instance Space. This way we can avoid having to build 
these objects from scratch. 

The Instance Space and the Library Space are implemented as data structures shared 
by the other active components shown in Figure 1. These components are implemented as 
separate parallel processes accessing the above data structures using the shared memory 
services of the host computer. This way only those which are necessary for the currently 
running simulation have to be loaded. These separate active components include: 

1. The Structure Editor which can be used to build objects either in the Instance Space 
or in the Library Space. 

2. The Display Interface which transforms the Instance Space objects into executable 
graphics primitives. The reason for introducing this interface was that the object 
representations of the Instance Space can be machine-independent, since the actual 
machine-dependent graphics format conversions will be done by this module. 

3. The File Interface provides the services to save and restore the objects in the 
databases. 

4. The Simulation Package operates on the entities in the Instance Space and performs 
operations similar to those of the ROBOSIM simulation library described previously 
(moving objects, collision detection, kinematics and dynamics calculations). 

5. The Simulation User Interface provides the features for the interactive control of the 
operations contained in the Simulation Package. 

6. The High-level Interface Package provides access to the services of the other active 
components from knowledge-based application programs, like task planners, etc.. 
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Many of the components described above have already been specified and are cur- 
rently being developed. Next we give a description of the internal details and current status 
of those components where such information is already available. 


3.3 Detailed Description of the Components 


The data structure specifications for the two shared databases were completed first, 
since they influence the design of the other components. Both data bases contain structures 
which can be allocated dynamically, and care was taken to ensure that the specifications do 
not contain built-in ’static’ limits regarding the number or complexity of the objects. There 
is an interface library associated with both shared data bases which provide unified object 
creation, access, etc., services for the other active components. 

The Instance Space contains the system’s working structural model object set. Some- 
times it contains redundant data if it speeds up the operation of the different active com- 
ponents. For example, the graphics display module can utilize the polygon list repre- 
sentation of a solid object most effectively, while the collision detection module operates 
on the edge list. In such cases we decided to store both (or all necessary) representation 
forms in order to gain as much execution speed as possible at the expense of a somewhat 
higher load on memory. The access library associated with the Instance Space contains 
services to create access and destroy objects, and there is an additional shape generator 
library (derived form the original ROBOSIM code), which can fill out these data structures 
with the graphical representations of the higher-level geometrical entities they represent. 
The Instance Space objects themselves do not contain information regarding their origin, 
only the data necessary to ’operate’ them. All objects are dynamically typed, and the 
operations ’know’ how to perform an action on the given type. Next we give some 
(simplified) examples about the data structures stored in the Instance Space: 

/** 

** ROBOSIM.NEW.H 
** 

** structure definitions for new ROBOSIM (draft version) 

*7 

typedef struct { 
double xx; 
double yy; 
double zz; 

} jjoint; 



typedef _point _bbox(2]; 
typedef double _trmat[4][4]; 


# define TP POINT 1 

# define TP~LINE 2 

#define TP~POLYLINE 3 

#define TP POLYGON 4 

#define TP~SOLID 5 

#define TPJOINT 6 

#define TP LINK 7 

#define TP'ROBOT 8 

#defmeTP ITEM 9 


#define TP"SCENE 10 

typedef struct { 

int type; 

_point coords; 

} o_point; 

typedef struct { 

int type; 

int color; 

_point start; 

_point final; 

} o_line; 

typedef struct { 

int type; 

int num_of_points; 

int color; 

_point pts[l]; /* actual number might change */ 

} o_polyline; 

typedef struct { 

int type; 

int num_ofjx>ints; 

int color; 

_point normal_dir; 

double normal_dist; 

_point ptsflj; 

} ojjolygon; 

typedef struct solid { 
int type; 

int color; 

struct _solid *next_solid; 

struct _solid ‘parts; 

_trmat ‘transf; 

_trmat *inertia; 

_bbox bounds; 

struct _object ‘origin; /* points to the editor structure 

which caused its creation */ 

int num_of_polygons; 
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o_polygon *polys[l]; 

} o_solid; 

#defme TP REVOLUTEJ 20 

#define TP"PRISMATIC_J 21 

typedef struct { 

int type; 

int subtype; 

_trmat jntparams; 

double llim; 

double hlim; 

} o joint; 

typedef struct { 

int type; 

int num_of_outputs; 

o_solid ‘shape; 
ojoint ‘injoint; 
o joint “outjoints; 

_trmat a_matrix[l]; /* actual number might change */ 

} o_lmk; 

typedef struct { 

int type; 

int num_of_links; 

o_link ‘linksflj; 

} o_robot; 

typedef struct { 

int type; 

union obj { 
ojine line; 

o_polyline plin; 

o_polygon poly, 

o_solid solid; 

o_robot rob; 

} ‘item; 

_trmat place; 

} o_scenepart; 

typedef struct { 

int type; 

int num_ofjtems; 

o_scenepart ‘parts; 

} o_scene; 

/** 

“ ALLOCATOR.H 
** 

“ definitions for the new ROBOSIM workspace memory allocator 
**/ 

# include "robosim.new.h" 
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init_memory(); 

free_all_objects(); 

trmat *make_trmatQ; 

o_point *make_point(); 

o_line *make_line(); 

o_p°ly!ine *make_polyline(/* int num_of_points */); 

o_polygon *make_polygon(/* int num_of_points */); 

o solid *make_solid(/* int num_of_faces */); 

ojoint *makejoint(); 

o_link *make_link(/* int num_of_outjoints */); 

o_robot *make_robot(/* int num of links */); 

o scenepart *make_scenepartQ; 
o scene *make_scene(/* int num_of_parts */); 

void free object(/* result of any of the above calls*/); 

/** 

** BUILDER.H 
** 

** definitions for the new ROBOSIM primitive object generator routines 

*7 

#include "robosim.new.h" 

ojwlygon *build_polygon(/* color, numpts,xl,yl, ... xN,yN */); 

o_solid *build_box(/* x,y^ */); 

o_solid *build_cylinder(/* r,h,numfaces */); 

o_solid *build_cone(/* r,h,numfaces */); 

o_solid *build_truncated_cone(/* rl,rh,h,numfaces */); 

o_solid *build_sphere(/* r,numfaces */); 

o_solid *gen_extrude(polygon, height); 

o_solid * gen_revsurface(polygon, angle, numfaces); 

void tr_translate(object,x,y,z); 
void tr_rotate(object,yaw,pitch,roll); 

r 

* remarks: 

* (1) all things are generated with their center of mass at the origin of 

* coordinate system. 

* (2) polygons are created in the XY plane 

* (3) if numfaces = 0 is given a default internal (size dependent ?) 

* value is used 

* (4) transformations are destructive, but extrude and revsurface not ! 

7 


The Library Space data structures contain a hierarchical description of the graphical 
object classes defined in the system. Since the possible most efficient operation is of a 
lesser concern here, these data structures are usually non-redundant. In contrast to the 
Instance Space entities, the higher level geometric concepts are also represented here (i.e. 



whether an object is a box or a cylinder, etc..) along with other properties (mass, etc..). 
Aside from these differences, the implementational considerations regarding this com- 
ponent are similar to the ones discussed at the Instance Space. Some examples of the 
entities stored in the Library Space are: 


r 

# rlib.h 

# R Library Space Datastructures 
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# define DEBUG 

/* Geometrical datastructures */ 

#deflne TypeTag short type; char subtype[2] 

/* Instances & calls */ 

#define RT SYMBOL 0 

# define RT - NUMBER 1 
#define RT~LIST 2 
#defme RT~MATRIX 3 

# define RT~OPER 4 
#defme RTASSIGN 5 
#define RT~SOLID 6 
#define RT"LINK 7 
#define RTJROBOT 8 

/* Definitions */ 

#define RT DSURF 10 
#define RT~DCOLOR 11 
#define RT~DSHAPE 12 
#define RT DSOLID 13 
#define RTDLINK 14 
#define RT~DROBOT 15 
#define RT DSCENE 16 

/* Link types - Subtype of LINK */ 

#define RT LFIXED 0 
#define RT~LPRISM 1 
#define RT'LREVOL 2 

/* Solid types - Subtype of SOLID */ 


#define RT PBOX 0 

#define RT~PCYLINDER 1 
#define RT PCONE 2 

# define RT PTRUNCATEDCONE 3 
#define RT~ PSPHERE 4 

#define RT~PEXTSURFACE 5 
#define RT'PREVSURFACE 6 
#define RT - PSOLID 7 



/* Operators - Subtype of OPER */ 

# define RT OPLUS 0 

# define RT _ OMINUS 1 
#define RT~OTIMES 2 

# define RT ODIV 3 

# define RT~UMINUS 4 

typedef struct symbol { 

TypeTag; 
char ’string; 

} Symbol; 

typedef struct number { 

TypeTag; 
float number; 

} Number; 

typedef union _value { 

Symbol symbol; 

Number number; 

} Value; 

typedef union _item Item; 

typedef union _expr Expression; 

typedef struct _operator { 

TypeTag;" 

Expression *argl, *arg2; 

} Operator; 

union _expr { 

Symbol symbol; 

Number number; 

Operator op; 

}; 


typedef struct _asgn { 
TypeTag; 

Symbol *lhs; 

Expression *rhs; 

} Assign; 

typedef struct _list { 
TypeTag; 

Item ’item; 
struct _list ’next; 

} List; 

typedef struct matrix { 
TypeTagf 
Value *values[4]{4]; 
} Matrix; 



typedef struct _color { 

TypeTag; 

Symbol ‘name; 

Number ‘red, ‘green, ‘blue; 
} Color; 

typedef struct _surface { 
TypeTag; 

Symbol ‘name; 

Number ‘red, ‘green, ‘blue; 
} Surface; 

typedef struct _shape { 

TypeTag; 

Symbol ‘name; 

List ‘params; 

Symbol ‘color; 

List *vars; 

Symbol ‘surface; 

List ‘points; 

} Shape; 

typedef struct _part { 

TypeTag; 

Symbol ‘name; 

Symbol ‘call; 

List ‘values; 

Symbol ‘color; 

Matrix ‘matrix; 

} Part; 

typedef struct _solid { 

TypeTag; 

Symbol ‘name; 

List ‘params; 

Symbol ‘color; 

List *vars; 

Number ‘nfaces; 

Symbol ‘surface; 

Number ‘mass; 

List ‘parts; 

} Solid; 

typedef struct Jink { 

TypeTag;" 

Symbol ‘name; 

List ‘params; 

Symbol ‘color; 

List *vars; 

Number ‘nfaces; 

Symbol ‘surface; 

Number ‘mass; 

Symbol ‘solid; 



List *values; 
list ‘matrices; 

} Link; 

typedef struct robot { 
TypeTag; 

Symbol ‘name; 
List ‘params; 
Symbol ‘color; 
List *vars; 
Number ‘nfaces; 
Symbol ‘surface; 
Number ‘mass; 
List ‘links; 

} Robot; 

union _item { 

Symbol symbol; 
Number number; 
List list; 

Matrix matrix; 
Value value; 
Assign assign; 
Part part; 

Link link; 

Robot robot; 


typedef struct _scene { 
TypeTagf 
Symbol ‘name; 
List ‘params; 
List ‘robots; 
List ‘solids; 

} Scene; 


typedef union 

^object { 

Symbol 

symbol; 

Number 

number; 

Value 

value; 

List 

list; 

Assign 

assign; 

Operator operator; 

Matrix 

matrix; 

Color 

color; 

Surface 

surface; 

Shape 

shape; 

Part 

part; 

Solid 

solid; 

Link 

link; 

Robot 

robot; 

Scene 
} ‘Object; 

scene; 


extern Object RlMalloc(); 
extern void RIFreeO; 
extern void InitTables(); 
extern void AddT ableEntry() ; 
extern Object RIMallocTypeO; 
extern Object MakeNumber(); 
extern Object MakeS ymbolQ; 
extern Object MakeColorDef(); 
extern Object MakeSurfDef(); 
extern List *MakeListItem(); 
extern Object MakeOperatorQ; 
extern Object MakeAssignQ; 
extern Object MakeShapeDef(); 
extern Object MakeBox(); 
extern Object MakeCylinder(); 
extern Object MakeConeO; 
extern Object MakeTruncatedO; 
extern Object MakeSphere(); 
extern Object MakeExtSurface(); 
extern Object MakeRevSurfaceO; 
extern Object MakeSolidCall(); 
extern Object MakePart(); 
extern Object MakeSolidDef(); 
extern Object MakeLinkDef(); 
extern Object MakeMatrix(); 
extern Object MakeLinkCall(); 
extern Object MakeRobotDef(); 
extern Object MakeRobotCall(); 
extern Object MakeSceneDef(); 

The Structure Editor is based on the interactive menu-based graphical editor 
developed for the ROBOSIM package. We plan to reuse most of the code developed 
during that work, but modified in order to operate on the data structures of the two shared 
data bases. An additional feature is that we want to preserve upward compatibility with the 
ROBOSIM language, so the editor will have a ROBOSIM source code interface. (But 
internally of course it uses different data structures). The Editor is currently under develop- 
ment. 


The Display Control module provides services used to control the graphical output of 
the system. (Viewing transformations, lighting, etc..). Parts of it are based on the already 
existing extended ROBOSIM code. The Display Module operates upon receiving messages 
from the Simulation Module regarding the changes in the working environment (i.e. which 
objects have been moved, etc..). It recalculates those of its internal descriptor structures 
(which are grahics hardware dependent - but all of these dependencies are localized within 
this module) which have been affected by the changes, and refreshes the screen. 



The development of the Simulation Library is a parallel effort here and in the old 
ROBOSIM environment, since many of the algorithms in it (collision detection, kinematics, 
etc..) are very complex, and even in the old version most of them are not operating on the 
original ROBOSIM data structures. For this reason we expect to use them in both versions 
with minor changes. 

The other components are still in the specification phase, so a more detailed descrip- 
tion is not possible at this time. 


3.4. Automation Interface for Structural Modeling Systems 

Previously we summarized the features we think are expected from a graphics struc- 
tural modeling system to utilize the power of knowledge-based techniques in three dimen- 
sional world modeling. The last part of this chapter describes some of these knowledge- 
based techniques themselves. The Department of Electrical Engineering at Vanderbilt 
University has a long history of building large knowledge-based engineering applications in 
the fields of instrumentation, process control, simulation and testing. In the course of this 
work we have developed numerous knowledge-based tools for this specific purpose. 

The design of large-scale engineering systems that must operate in unstable, changing 
situations is one of the foremost challenges of the information sciences. Conventional 
design methodologies are based on the availability of a priori information about the en- 
vironment and the system to be observed and controlled. The information is expressed in 
the form of models representing relevant aspects of the environment. The basic modeling 
principles of the system sciences such as separation, selection, and model economy [1] are 
the key approaches for managing complexity. The essence of these principles is 
simplification until a model of manageable size is obtained. By imposing constraints on the 
possible behavior of the environment, the analysis and/or synthesis of the corresponding 
automation system becomes feasible. 

There are two main ways how knowledge-based techniques can be used to satisfy the 
above goals. In many cases the more traditional rule-based, shallow modeling techniques 
can provide quite satisfactory results. The other approach is to use as much structural 
information about the environment as possible, in order to create a structural, deep model 
of the system. Both approaches have advantages over each other, so the best strategy is to 
use them together to solve complex engineering problems. 



The graphics modeling toolkit described previously is intended to be used together 
with knowledge-based controllers using either one or both of these techniques. We plan to 
use the NASA-developed CLIPS expert system shell as the vehicle to build the rule-based 
parts of the intelligent controllers. This choice was influenced by factors such as the easy 
availability of the CLIPS system, its good performance (due to the fact that it has been 
implemented entirely in C), and the easy portability of it. For the intelligent controllers 
using structural, deep modeling techniques, we plan to use the MULTIGRAPH program- 
ming environment (developed at Vanderbilt) described later in this section. 

We think that the two knowledge-based techniques can ’peacefully coexist’ in com- 
plex systems using geometric, structural modeling. For example, in one of the intended 
application areas, in Space Station automation, a typical scenario for the joint usage of the 
different techniques might be the following: 

1. Application areas for geometric modeling techniques (ROBOSIM or the new 
package): 

- The geometric model of the Station itself 

- Models of different manipulators operating on the outside or in the inside of 
the Station 

- Other moveable attachments to the Station, like solar panels, hatches, etc.. 

2. Application areas for rule-based techniques (CLIPS): 

- Scheduling of different operations on the Station 

- Task Planning for robotics applications on the Station 

- Creating qualitative models of those subsystems which can not modeled 
analytically due to their complexity or lack of information 

3. Application areas for structural modeling techniques (MULTIGRAPH): 

- Modeling those subsystems where the structural and operational data is 
available to create qualitative, structural models 



- Modeling control systems 

- Fault propagation modeling and failure analysis 

Of the above three techniques, the geometrical structural modeling toolkit has al- 
ready been described in this report, and the rule-based techniques are supposed to be well- 
known, since they have been in use for quite a long time. But we think, that the structural 
knowledge-based modeling methodology and its run-time environment (the MULTI- 
GRAPH architecture, which has been developed at Vanderbilt), deserves some more 
explanation. 

Model-based knowledge-based methodologies have great potential in implementing 
automation systems for a wide range of applications. The main idea is quite straightforward 
and includes the following steps. 

- A dynamic model of the environment (the system to be observed or controlled) is 
included in the higher-level knowledge-based controller of the automation system. 

- The model is continuously updated based on observations. 

- The control system is modified (structure and parameters) if state changes in the 
model require it. 


We will focus on the computational problems of creating structurally adaptive control- 
lers by using model-based techniques. The purpose of the discussion is to show the key 
components of a programming and execution environment that can be used for implement- 
ing this new system category. 

The main computational requirements in the implementation of structurally adaptive 
controllers are the followings: 

- The dynamic model of the environment and its interactions with the structure of the 
control system must be represented. 

- The representation must be used as part of the control process, i.e. changes in the 
environment model must be mapped into changes in the structure of selected 
automation system components. 



- The structural changes must be executed without suspending the system operation. 

By using artificial intelligence terminology, the first requirement creates a knowledge 
representation problem. Naturally, the model-based approach demands the explicit repre- 
sentation of automation system models. The key issue is what kind of representation 
techniques can be used for this purpose? The second requirement addresses the problem of 
knowledge utilization. The knowledge which represents the interactions between the environ- 
ment and the structure of the control system has to be actively used for modifying the 
system operation. The problem is how to "convert" this knowledge dynamically into im- 
plementation specific terms? The third requirement is closely related to the computational 
model used in the execution environment of the control system. The question is what kind 
of computational model can support the dynamic reconfiguration of a processing system in 
execution time? 

The main difficulty in the technology of intelligent adaptive automation systems is 
that realistic implementation can not be built without finding satisfactory solution for each 
of these problems. In the followings we will focus on the description of the components of 
the Multigraph Architecture which has been designed to serve as a generic programming 
and execution environment for this system category. 

The Multigraph Architecture (MA) has been developed for building a broad category 
of intelligent systems operating in real-time environment. The MA has been used as a 
framework for intelligent instrumentation, automatic test configuration, and process 
control systems. The basic layers of the MA are the: (1) hardware layer , (2) system layer, (3) 
module layer, and (4) knowledge layer. In Figure 2, the three main levels of the MA are 
shown from the user’s point of view. 

- Model Designer. The design and implementation of model-based, intelligent control 
systems requires extensive modeling. Because the unforeseen operational conditions 
might require structural modifications in the control system, the models must be 
hybrid. Hybrid models explicitly represent not only quantitative, but qualitative, 
structural attributes of the environment and the control system. Model designers 
must be supported by appropriate tools to build and validate these models. 

- Application Programmer. The models that are used in the design and implementa- 
tion of intelligent automation system are domain specific by their very nature. The 
form of the models (concepts, relationships) are different in chemical processes, 
mechanical processes, information processing systems etc., because the models must 
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Figure 2. Structure of the Multigraph Architecture 

reflect the selected properties of these systems. However, some of the basic model- 
ing principles, such as composition techniques, organization in levels of abstraction, 
multiple-aspect representation, etc. are quite universal. This generality makes it 
possible that the creation of domain specific modeling tools can be supported by 
general methodologies. The application programmer level in MA includes those 
components that are used for building various, domain specific modeling environ- 
ments. 

- System Programmer. The lowest level of MA provides interfaces to the components 
of the Multigraph Execution Environment (MEE). The central element of MEE is 
the Multigraph Kernel (MK), which is the run-time support of the Multigraph 






Computational Model (MCM). MCM is a macro-dataflow model which satisfies the 
required dynamic behavior mentioned before. 

The models that are created during the modeling process are complex structures 
representing different aspects of the environment, the control system and their interactions. 
It is important to note that in these models the structural complexity is the dominant factor, 
the algorithmic complexity is typically negligible. This fact had deep influence on the 
properties of the Multigraph Programming Environment (MPE). The two basic techniques 
used for supporting this activity are (1) multiple-aspect model building and (2) declara- 
tive/graphic programming. 

- Multiple-aspect model building. Characterization of objects from different aspects is 
a well known method in modeling. There are artificial intelligence (AI) tools that 
directly support the creation of "multiple views". According to our experiences, the 
real difficulty is not the representation of different aspects but the expression of the 
interactions among them. The critical question is how to facilitate the well struc- 
tured representation of these interactions? MPE allows the declaration of 
structurally independent (SI) and structurally dependent (SD) modeling aspects. 

- Declarative/graphic model building tools. Modeling requires tools for representing 
the models. The representation technique has to satisfy two contradictory require- 
ments. First, the representation system must provide "interface" for the model 
designer, i.e. the represented model has to be easily comprehensible by humans. 
Second, the represented model has to be machine readable, because the models 
constitute the "knowledge-base" which determines the system operation. Based on 
these requirements and on the fact that the models express dominantly structural 
information, MPE supports two equivalent representation form: declarative 
languages and the corresponding graphic representation. The model building process, 
which is performed by the model designers is fully graphical and directly supports 
SD and SI modeling. 

Pictures 6 and 7 show the graphic model a reconfigurable controller for a 
simple robot arm. The arm is controlled by (a) a proportional controller, or (b) a 
PAD controller. The reconfiguration occurs when the "Checker" finds the perfor- 
mance of one of the controller unacceptable. The figure shows only the top level 
structure of the controllers and the simulation model of the arm. Each of the boxes 
have an internal structure on the lower levels of the hierarchy. The graphic model 
has been built by using the iconic editor of MPE. The pictures also show parts of the 



equivalent declarative language representations of the model. The declarative 
language is a variation of the "frame languages", which can be easily defined for the 
different modeling domains. 

- Test and Validation Tools. Declarative languages offer excellent opportunity for 
automatic test and validation. The basic approach used in the test and validation 
toolset of MPE includes the following steps: 

1. the declarative language forms are mapped into a unified graph structure, 

2. test and validation criteria are defined for the different modeling aspects, 

3. the criteria are expressed as graph properties, and 

4. graph algorithms are used to check the properties. 

The methodology supports the automatic consistency testing of the individual 
modeling aspects and the consistency testing among the SD aspects. A serious 
limitation of the test approach is that only static properties of the models can be 
tested this way. In a new research direction we address the problem of testing the 
dynamic, run-time behavior of the system. 

An important goal of MPE is to facilitate the definition of declarative languages and 
the corresponding graphic editors for new application domains. Generic tools belonging to 
the level of the Application Programmer support this task which includes the following 
steps: (1) definition of the syntax of the declarative languages, and (2) configuration of the 
corresponding graphic editor. The two programming tools developed for this purpose are 
the Declarative Language Language (DLL), and the Programmable Graphic Editor 
(PGE), respectively. 

Multiple-aspect models of the external environment (platforms, signal sources, etc.), 
the various components of the control system (monitoring systems, controllers, etc.), and 
their interrelationships embody the information that is necessary to generate a specific 
instance of the knowledge-based controller for the automation system. The problem of 
system integration is to generate this instance from the models, or in other words, to map 
the models into an appropriate executable program. Because of the implementation 
method of this mapping, we will call this process model interpretation. 
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Pictures 6a-b: Graphic model of the Reconfigurable Controller with Declarative 

Language Representation 
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The complexity of the model interpretation process largely depends on the nature of 
the models. If it includes only the symbolic, static model of a specific system, e.g. the model 
of a controller, the model interpretation process is reduced to the complexity of simple 
application generator systems. In the general case, the structurally adaptive controllers 
require the following capabilities from the model interpretation process. 

- Multiple-aspect interpretation. The result of the model interpretation process must 
generate more than one subsystems. Multiple-aspect model interpretation means 
that the mapping process must interpret the models from the aspects of the various 
subsystems to be generated. 

- Decision making. The complexity of the mapping process is largely the consequence 
of the fact that the models are not structured according to the subsystems of the 
system to be generated. (Except the simple application generator problems, where 
modeling is usually constrained to specific computation systems to be generated.) 
Indeed, in model building time the natural way of thinking is to focus on selected 
aspects of the environment, the control system and their interactions without any 
explicit considerations to the actual way of implementation. The model interpreta- 
tion process has to be "smart enough" (1) to collect the relevant information from 
the models for the various subsystems, and (2) during this process to make decisions 
on the actual structure of the computation system by analyzing the interaction of the 
different modeling aspects. 

- Dynamic behavior. The essence of any structurally adaptive system is the capability 
for dynamic reconfiguration of subsystems after a change in the working environ- 
ment has been detected. It means that the model interpretation process has to be 
restartable from that point which has been effected by the detected change. 

These capabilities required the elaboration of a special computation model in the 
Multigraph Execution Environment (MEE). MEE provides a system integration tool by 
supporting the dynamic configuration of application programs from a library of precom- 
piled elementary processing modules. This configuration process can be performed by the 
higher-level knowledge-based system components using an appropriate builder interface of 
the MEE. Frequently the usage of the MEE also enables the utilization of the inherent 
structural parallelism in the application programs, since it is quite typical that many of the 
processing modules of an application configured using the above method can be executed 
concurrently, provided that the underlying hardware architecture supports this. 



MEE uses a macro-dataflow model as its basic computational model. The reasons for 
this choice were (1) the well-known nature of the dataflow computations due to the sig- 
nificant amount of research conducted on exploring the theoretical properties and im- 
plementational issues of these, and (2) the fact that many engineering system models (for 
example the signal flow graphs used in signal processing and process control systems) can 
easily be mapped into dataflow graphs. Some extensions were added to the "typical" 
dataflow computational concepts, because the MEE serves as a unified run-time support 
for the different parts of the intelligent automation systems, and these parts might use 
different models of computation (for example signal-flow graphs, discrete event simulators, 
rule interpreters, constraint propagation networks, etc..). 

The applications in the MEE are mapped into a control graph. A control graph in the 
MEE is defined by its actomodes, datanodes and connection specifications. The actomodes 
are the active components of the graphs. They execute an application module (the script) 
which can be written either in Lisp or in other non-symbolic languages (C, Fortran, Pascal). 
The scripts are position independent, they communicate with the other graph components 
using the communication primitives of the MEE and the ports attached to the actor node. 
If the code of the script is reentrant, it can be attached to several actomodes. The MEE 
provides a way to pass a local parameter structure to the scripts, which is called the context 
of the actomode. Beside the typical dataflow control principle (a node can be fired when- 
ever all of its inputs are present - ifall mode) MEE also supports another mode of actor- 
node execution, where a single input data is enough to fire a node ( ifany triggering mode). 

The datanodes are the passive components of the control graphs. Their function is to 
store the data generated by the actomodes. They can store either a stream of data, or only 
the last data sent to them. 

MEE supports several operation modes of a control graph. A graph can be operated 
either in data-driven or demand-driven mode, or in a combination of the two modes. In the 
data-driven mode, the data sent to a datanode propagates a control token to the following 
actomodes, which will fire after collecting the necessary tokens. The demand-driven mode 
means that an attempted read operation on an empty datanode will send a request token to 
all possible sources (ie. the connected actomodes) of the information. 

MEE provides an environment and task structure which is used to assign the various 
system resources of the system hardware and software (processors, tasks, special hardware 
units, etc..) to the execution of the actomodes in the computational graphs. 



The structure of a typical implementation of the MEE can be seen in Figure 3. MEE 
can be depicted as a set of protected data structures which can be accessed through the 
following three interfaces: 



Figure 3. Structure of the MEE 


- Module Interface: which provides the data and request propagation calls for the 
application modules attached to the actomodes. 

- System Interface: which is responsible for scheduling the elementary computations 
using the system resources provided by the host operating system. 

- Builder and Control Interface: which provides the control graph building and execu- 
tion control facilities for the higher-level knowledge-based system components. The 
services of this interface can operate on an already active computational graph, 
which enables the dynamic reconfiguration of the application programs. 






MEE offers a set of debugging tools which are especially helpful in concurrent sys- 
tems. These include a stepper/tracer facility and a graphic monitor, which generates and 
displays the graphic layout of selected parts of the control graph, and dynamically displays 
the status of the nodes in the graphic window. 

The computational model and the details of its implementation were selected such 
that the Kernel can provide the same execution environment on a variety of computer 
architectures, by hiding the details of the (possibly parallel) execution from the application 
modules, which can be simple sequential procedures in every case. 



4. CASE STUDIES 


The modeling methodologies and tools described in the previous chapters provide a 
usable working environment for testing automation concepts regarding space applications. 
This chapter describes some of this (planned and already completed) work. First a struc- 
tural, geometric model of the Space Station is presented, which was prepared using the 
graphical modeling techniques of Chapter 2. Next some planned modeling efforts are 
described which combine this structural model with knowledge-based techniques to simu- 
late various operational aspects of the Station. Part of this work is the modeling of the 
Space Station Environment Control and Life Support System (ECLSS), which has already 
been performed using the symbolic modeling techniques introduced in the previous chap- 
ter. 


4.1 Space Station Modeling Using ROBOSIM 


In the last three years, ROBOSIM has been applied in numerous occasions to 
develop and study real-time models of industrial manipulators. It’s use, however, was not 
limited to robotics only. Recently, ROBOSIM was put to use to support a sequenced build- 
up of the space station model. The porting of ROBOSIM to a real-time graphics worksta- 
tion, the HP 350SRX, with it’s 3D graphics capabilities, knobs and menus served as a more 
interactive and user-friendly tool which allowed for superior illustration and detailed 
examination of different parts of the space station model. 

In designing the space station, just like in designing a robot, the selection of the 
robot’s kinematic design is usually considered first. The number of robot joints, type of 
joints(rotational, sliding or fixed) and the physical configuration are all important factors of 
the robot’s kinematic design. 

After careful analysis of NASA’s latest configuration of the space station model (SS, 
for short) and knowing ROBOSIM’s capability of handling multiple number of 



manipulators within the same working plane, a modular approach was chosen to construct 
the SS model. 

The SS model was broken in to several independent, serially linked manipulator 
models, all assigned the same reference frame. Each manipulator consisted of separate 
parts, where each part was built as a compound object made of primitives, such as 
boxes, spheres, cylinders and user-defined shapes. These parts were then assigned the 
correct kinematic parameters and mass properties and finally assembeled together using 
ROBOSIM. 

The modular approach was a necessary approach as well as a practical one. It was 
necessary, because it helped overcome the problem of serial-linkage, usually associated 
with robotic simulation packages, where a movement in one link will cause a movement in 
the next. 

Breaking down the model into separate independent manipulators, helped overcome 
this obstacle. For example, each set of the solar panel assemblies could now be adjusted 
and controlled independently of the other set. The modular approach is also practical 
because it allowed for complicated models to be created in smaller parts and assembled as 
the designer required. Changes could then be made to any component of the model 
without affecting other parts. New parts or manipulators could also be added just as easy 
without affecting any of the existing models. 

The SS model was broken into five independent, serially linked manipulators, with 
each manipulator representing a desired set of rotations and/or translations. These models 
were represented as follows: 

1. Two solar panel structures (Pictures 8 and 9), each of which is treated as a separate 
manipulator attached to the side of the main truss assembly. Since both solar panel 
structures were physically identical, only one structure had to be constructed. The 
other was simply replicated, but assigned different kinematic properties. This 
feature of ROBOSIM helped save time and effort, since structures can be saved in a 
file for later usage. 

2. Two identical sets of solar panels, heat radiator and truss assemblies, each treated as 
an independent manipulator. These assemblies attach to both ends of the middle 
truss assembly. Each assembly has two rotational movements. One for the solar 
panels, to position them in a direction facing the sun, the other for the whole as- 


sembly structure to be able to position the heat radiators away from the sun. 

3. Mobile servicing robot, with five rotational joints, sliding on a set of rails to be at- 
tached to middle truss assembly (Picture 10). The robot was modeled as a six degree 
of freedom manipulator, with the sliding rails serving as a translational joint. The 
robot is used to perform routine tasks, e.g., inspection and maintenance. 

4. Finally, the middle truss assembly was built. It included crew living modules, antennas 
and truss assemblies all attached together to create the main body, to which all 
other sub-models attach (Picture 11). A common frame, to which all other 
manipulator models refer, was assigned at the middle of this truss assembly. 

5. For easier debugging, this final structure was broken into three parts: Crew-living 
modules, antennas and trusses. A set of two non-shaded, user-defined cubes were 
built and propagated, using temporary storage registers, to construct the truss 
assembly. This illustrates ROBOSIM’s ability to create user-defined shaded and 
non-shaded objects as parts of the same model. 

With the links of all five models being defined and a common reference frame as- 
signed, the graphics display program was used to assemble the different parts, in a pre- 
assigned configuration, to generate the desired SS model (Picture 12) The menu box, top 
right of the screen, provides various options with which the user can interactively view and 
control separate parts of the model. 

Separate routines could also be linked to the Graphics display program to assign joint 
limitations and/or set motion along any parametrically defined functions. Two sets of 
predefined motion for the main solar panel assemblies is shown in Picture 13. 


4.2 Operational Modeling of the Space Station 


Space Station automation requires the analysis of the complex material, energy and 
information transfer processes from many different aspects. The structural model intro- 
duced previously is just a representation of one of these aspects, but to cover the full range 
of possible operations, it has to be combined with other models representing the different 
aspects and using different modeling techniques. The integrated modeling environment 
which was the subjects of the previous parts of this report offers a unique opportunity to do 
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Picture 8: Secondary solar panel structure 
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Picture 9: Main solar panel assembly with radiators 



this. Below we list a couple of the problems which are well suited for this approach. 

Attitude Control System and its Dependencies: The Space Station is a large structure, 
which due to the different disturbing effects (solar wind, etc..) requires a constant control 
of its attitude. This is done by a triple gimbaled gyrator system (according to the plans). 
The structural model of the station together with the (already existing and newly 
developed) elements of the Simulation Library could provide a toolkit to test the orbital 
mechanics and the attitude control problems related to the station. 

But this is just one of the aspects of the attitude control problem! The Space Station 
is a relatively small closed system, so everything influences everything. Normally the triple 
gimbaled gyroscopic attitude control system is sufficient to control the orientation of the 
Station. But during the course of the operation, the rotational angles of gyroscopic wheels 
might reach a position where they align with each other - which means that the system is 
not capable of control any more. In such cases the gyroscopes must be ’recharged’ i.e. their 
angles of rotation made (approximately) perpendicular again. This of course will offset the 
orientation of the Station which then must be corrected using thrusters. It is expected that 
this operation will have to be performed at about every tenth orbit. There are several 
constraints which influence this: 

1. This ’recharging’ operation might disturb some ongoing low-gravity experiments 
(because it introduces relatively high accelerations), so these have to be considered 
when scheduling it. 

2. There are other orbital maneuvers which affect the attitude control (docking or 
launch of objects). A higher-level controller which schedules the recharging ac- 
tivities of the attitude control system must know about these events too. 

3. While the Station is on the ’sunny’ side of the Earth, the photelectric cells should be 
operating at the possible highest capacity. If the sudden changes in the station’s 
orientation can not be followed by the control system of the panels then the energy 
production might suffer. On the other hand the operation of the solar panel’s 
alignment mechanism itself influences the attitude control system. 

The Electrical Energy Production and Distribution System itself is an interesting area of 
study, due to the limited energy supply and the interactions between the different con- 
sumers. Some of the problems in this area are: 
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Picture 10: Mobile servicing robot with slidin g rails 
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Picture 11: Middle truss assembly with antennas and crew modules 





Pictures 12a-b: Full and close-up views of the complete Space Station model 
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Pictures 13a-b: Views illustrating different rotations of main solar panel assemblies 



1. A control system must be developed which utilizes the periods while the Sun is 
visible most efficiently by aligning the solar panels as close to perpendicular to the 
Sun as possible. We have already begun developing a model for such a control 
system for this purpose utilizing the structural model of the Station and some of the 
higher-level symbolic tools introduced previously. 

2. A higher-level controller of this subsystem must predict the future energy produc- 
tion (interactions with attitude control and other orbital operations!), and based on 
the reserve energy and projected production must schedule the operation of the 
different consunmers. This seems to be a task to be solved using knowledge-based 
techniques, possibly by using the modeling techniques of Chapter 3 to simulate the 
different consumers. 

3. There are vital subsystems on the Station whose energy demands must be satisfied. 
An example of these is the Environment Control and Life Support System (ECLSS). 
Beside being a very important energy consumer, ECLSS is also a big energy con- 
sumer. If it is predicted that ECLSS’s energy demands can not be met, the whole 
operation of the Station may have to be rescheduled. Actually ECLSS itself is a set 
of interrelated subprocesses, some of which are not as important as the others. For 
example in the case of an energy shortage the air control subsystems for the experi- 
ment modules might operate at a reduced capacity, while it is not true for the crew 
modules. Such a decision will result in having to stop some of the ongoing experi- 
ments. But this again is just one of the possible interrelations! 

A common characteristics of the above examples is that modeling them requires 
considering many aspects of their operation. Some of these aspects can be expressed in 
quantitative terms, while others only in qualitative ones. This fact is the best justification 
for an integrated automation simulation and modeling testbed, containing (1) geometric, 
graphical modeling tools for spatial modeling of the different systems of the Station (e.g. 
ROBOSIM), (2) a rule-based programming environment for creating expert controllers 
and qualitative, knowledge-based models (e.g. CLIPS), and (3) tools for creating deep, 
structural, knowledge-based models for adaptive control and failure analysis (e.g. MULTI- 
GRAPH). 



4.3 Study of the Space Station ECLSS 


One of the most important systems of the Space Station is the Environment Control 
and Life Support System (ECLSS). This is a vastly complicated system with many interact- 
ing subsystems. Design of low-level control systems for these subsystems is based on model- 
ing the process dynamics. Development of the diagnostic system requires the elaboration of 
sophisticated fault models, and the construction of the operator interface is closely related 
to various qualitative models of the subsystems. The analysts can develop these models of 
different levels of abstractions, and can apply them for a particular purpose. But how can 
we secure the consistency of the models if they are developed separately? How can we 
ensure that a subsystem to be designed will be synergic with the related process models? 
How can we validate the models? 

Due to the difficulty of these problems, the support of modeling is of paramount 
importance in a simulation testbed for automation. The purpose of this case study is to 
demonstrate the use of a multiple-aspect modeling technique in analyzing the diag- 
nosability of the ECLSS. The study is being conducted in close cooperation with the Boeing 
Aerospace Company, Huntsville, Al. Since the task has not been completed yet, we 
describe only partial results of the ongoing modeling effort. 

Objectives of the ECLSS study 

ECLSS is a large system comprising complex material, energy and information 
transfer processes. The primary tool for the design and operation of the system is extensive 
modeling. The models help to understand the ECLSS in the design phase, and they are the 
key components of the monitoring, diagnostics and control system in operation time. 

From a methodological point of view, we consider the ECLSS design process as an 
incremental model building activity, in which various system components are defined in 
terms of specific models. The design is successful if the individual models are correct, and if 
the various models are consistent with each other. If the progress in the design process is 
represented in the form of a set of formal models (quantitative and qualitative), the inter- 
mediate results can be tested and validated by using the following techniques: 

1. The consistency of the models of different levels of abstraction can be tested by 
using mapping rules among the modeling aspects. 



2. The models can be used for the generation of quantitative/qualitative simulations of 
the system, in order to test its expected behavior from a selected aspect. 

3. The performance of specific subsystems (e.g. diagnostics, or control) can be tested in 
a simulated environment. 

AI provides a rich selection of modeling techniques that can support this process. 
Knowledge representation techniques can be developed to describe qualitative and quan- 
titative features of systems. These representations can be used to test the correctness of the 
individual models, to check the consistency among the related modeling aspects, and to 
analyse different features of the system designed. 

The objective of the study was to test the diagnosability of the ECLSS and provide 
advice on optimum sensor allocation. The specific objectives are the following: 

1. Multiple-aspect modeling of ECLSS. The models will define the energy, material 
and information processes in the system in a hierarchically organized way. These 
models include the Hierarchical Process Models (HPM) which serve as the dominant 
modeling aspect for the study. HPM provides the context for other, dependent model- 
ing aspects. The structure of the physical processes in ECLSS are modeled by using 
the graphic/declarative modeling techniques of MPE. 

2. Hierarchical Fault Models (HFM) of ECLSS. The fault models specify fault modes 
and fault propagation paths. The structure of the fault models corresponds to the 
structure of the process models, since faults can propagate only through physical 
interactions that are expressed in the process models. The multiple aspect modeling 
methodology of MPE ensures the consistency between the process models and the 
fault models. 

3. The issue of diagnosability can not be separated from the diagnostic method to be 
used. A sophisticated, model-based diagnostic system which can localize fault 
sources by analyzing detected alarms will be applied for the analysis. This diagnostic 
method can reveal how the sensor placement influences the diagnosability of the 
ECLSS. 

Although, the study is limited to the issues of diagnosability, we can easily expand the 
system later with other modeling aspects, such as modeling the monitoring system, operator 
interface, control system, etc. By using the automatic program generator services of MPE, 



the models can be used for generating an executable version of these sub-systems. 
Model-based diagnostic system 

The problem of diagnosability can not be separated from the diagnostic technique to 
be used during system operation. In the ECLSS study we have used a sophisticated model- 
based diagnostic system, which applies a hierarchically organized fault propagation model. 
In this section we summarize the properties of the diagnostic system and discuss the 
specification of the fault model. 

A real-time fault detection and diagnosis capability is absolutely crucial in large-scale 
space systems. Some of the existing Al-based fault diagnostic techniques like expert systems 
and qualitative modeling are frequently ill-suited for this purpose. Expert systems are often 
inadequately structured, difficult to validate and suffer from knowledge acquisition bot- 
tlenecks. Qualitative modeling techniques often generate a large number of failure source 
alternatives, thus hampering the speed of the diagnosis. 

In this study we use a graph-based technique which is well suited for real-time fault 
diagnosis. A Hierarchical Fault Model of the system to be diagnosed is developed. At each 
level of hierarchy, there exist fault propagation digraphs denoting causal relations between 
failure-modes of subsystems. The edges of such a digraph are weighted with fault propaga- 
tion probabilities and fault propagation time intervals. Efficient and restartable graph 
algorithms are used for on-line, fast identification of failure source components. 

Requirements for the diagnostic system 

A real-time fault diagnostics system has to function in an environment where new 
alarms may constantly be generated, due to the propagation of failures. To cope with such 
a time-changing scenario the diagnostics system must have the following characteristics: 

1. Signal Processing, Alarm Generation and Failure Source Identification software 
must be as fast as possible. The first two are usually standard well-defined and 
analyzed algorithms, and hence, virtually all speed improvements have to be 
achieved in the failure source identification phase. 

2. The diagnosed results must be updated as time elapses and new alarm information 
is received. These results must be accurate but need not have a fine resolution. This 
implies that in the early stages of diagnosis a large component such as the Potable 


Water Assembly can be identified as the fault source. The resolution of this fault 
source is further refined with the passage of time and additional alarm information 
to a unique component inside the Potable Water Assembly. 

3. The User-Interface must present the current status of diagnosis in a comprehendible 
manner, reflecting the level and the granularity of the system under diagnosis, at 
which the diagnostics system is operating. 

Graph-based approach in fault diagnostics 

The basic philosophy of the graph-based approach is based upon multiple-aspect 
modeling. The system under consideration is hierarchically decomposed from many aspects 
in order to yield a set of different models. The functional decomposition leads to the Hierar- 
chical Process Model (HPM) and a structural decomposition leads to a Hierarchical 
Physical Component Model (HPCM). A Hierarchical Fault Model (HFM) is developed in 
the context of HPM with links to the HPCM. 

The technique of hierarchical decomposition is widely used during model building for 
the following reasons: 

1. Design, knowledge acquisition, and knowledge-base maintenance of large complex 
systems becomes structured and easier. 

2. Running the same graph algorithms on smaller number of nodes many times takes 
lesser time than running them on the entire set of nodes in a system. For example it 
takes a longer time to run an 0(n ) algorithm on a graph with 200,000 nodes than it 
takes to run the same algorithm 200 times on a graph with 100 nodes. 

3. It is possible to conduct the search for the failure source on the HFM in a parallel 
manner, thus enabling speedy diagnosis. 

4. In most cases a large granularity component assembly can be identified as a failure 
source at an early stage, and then the search needs to proceed only in that com- 
ponent’s part of the model. 

Hierarchical Process Model (HPM) and Hierarchical Fault Model (HFM) 


A process in the HPM can be thought of as a functional unit carrying out a specific 



function in the system, by utilizing different physical components. Different processes on 
the same level may interact with each other through shared physical components. Processes 
in the HPM can be associated with many different components in the HPCM as shown in 
Figure 4. In the context of each process the following are acquired: 




Figure 4. A Hierarchical Process Model 

1. Process Failure-Modes. 

2. Process Alarms and alarm-generators. The alarm-generators accept sensor inputs 
and if needed, generate the appropriate alarm. 
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3. Alarm Failure-Mode associations. 


4. Failure-Mode Physical Component associations. 

Each process in the the HPM has its fault model, therefore fault models are con- 
sidered to be dependent aspects to the process models. This model is determined by the 
failure-modes of the process, and if present, the failure-modes of its subprocesses. All these 
failure-modes form nodes of a fault propagation digraph, with directed edges between 
individual failure-modes signifying a fault propagation possibility. Each edge in this graph 
is weighted with two parameters a fault propagation probability and a fault propagation 
time interval in terms of a minimum and a maximum. The fault propagation digraph of a 
process on level i is shown in Figure 5. The collection of all such fault propagation digraphs 
and failure-mode physical components associations results in the HFM. It is possible to 
extract the basic structure of the fault propagation digraph from the process models, since 
most faults can only propagate along physical connections. 

Diagnostic Algorithm 

By analyzing the fault propagation digraph together with detected alarms, the pos- 
sible failure sources can be found. This process can be migrated to lower levels of process 
hierarchy in order to get a better resolution. The failure source identification process 
consists of two algorithms: the Failure Source Process Identification (FSPI) and the Fault 
Source Component Identification (FSCI). An Inter Level Migration (ILM) process per- 
forms the task of searching the process hierarchy for the best resolution of the possible 
faulty source component. 

The FSPI algorithm obtains as input the fault-propagation digraph of a process to be 
diagnosed. It also receives all alarms currently ringing within that process and its sub- 
processes. This algorithm is accurately capable of detecting under most circumstances, the 
occurrence of either a single or a multiple fault in the process. On completion, this algo- 
rithm returns the possible fault source subprocesses and their fault source failure-modes. It 
uses the following constraints to determine the fault source in case of a single fault condi- 
tion : 

1. Reachability Constraint : All ringing alarms shall be reachable from the detected 
source failure-modes. 

2. Monitor Constraint : No failure-mode with a normal alarm shall lie on a path from 
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Figure 5. Fault Propagation Digraph of a Process 

any of the detected source failure-modes to any of the failure-modes with a ringing 
alarm. 

3. Temporal Constraint : All ringing alarms shall be individually reachable from the 
detected individual source failure-modes within the time interval computed from 
the time intervals found on shortest path between each individual alarm and source 
failure-mode pair. 

4. Consistency Constraint : There shall be no failure-mode with a ringing alarm whose 
reachability time from a source failure-mode is greater than the maximum 



reachability time of a failure-mode with a normal alarm from that detected source 
failure-mode. 

The algorithm is closed and complete and is thus suitable for fast location of failure 
source processes. 

The FSCI algorithm takes as input a list of detected source failure processes and their 
source failure-modes. In case of a single fault condition it returns a union of all physical 
components associated with the source failure-modes. In case of a multiple fault condition 
it tries to find a common component amongst all the source failure-modes. If successful! it 
returns that common component, and if not it returns a union of all associated components. 

The ILM process detects the highest level of the process in which alarms are ringing. 
It then tries to search for a failure source by running the FSPI and later the FSCI algo- 
rithms on all processes in that level. The results are used to guide a breadth-first search of 
all processes present in the next lower level. This process continues until the lowest level of 
hierarchy is reached. At this point the best possible resolution of the failure source is 
achieved. If during this migration an alarm rings in a higher level than the current one 
under processing, the ILM goes to that higher level and restarts the diagnosis. At any point 
in time the ILM can present its best guess of the failure source in any level of process 
hierarchy. 

Diagnostic System Architecture 

The Real-Time Fault Diagnostic System required: 

1. the potential use of a distributed computing architecture, 

2. support for a parallel programming model and 

3. integration of symbolic and numerical computations. 

The diagnostic system architecture is shown in Figure 6. A Monitoring sub-system 
handles the job of acquiring sensor outputs and alarm-generation. The Diagnostic sub- 
system consists of the diagnostic manager, diagnostic methods and a display manager. The 
diagnostic manager accepts as input all generated alarms and is in charge of conducting the 
inter-level search for the failure source. During this search it may send a process to the 
diagnostic methods object asking it to perform either the FSPI algorithm or the FSCI 



algorithm. The diagnostic methods perform the requisite algorithm and reports the result 
back to the diagnostic manager. These results are used by the diagnostic manager as a 
guide in its search. As soon as results are obtained for a level in the hierarchy they are sent 
over to the display manager for displaying them to the user. 



Figure 6. Diagnostics System Architecture 


Research Plan for the ECLSS Analysis 

The analysis is being conducted in parallel with the ECLSS design activity. The 
information for the ECLSS models is being acquired from BAC design engineers. The 
main steps of the analysis are the following: 





1. Definition and refinement of the HPM and HFM for the ECLSS. 

2. Derivation of a real-time alarm pattern simulator from the HFM. The alarm pattern 
simulator generates alarm sequences from the HFM by using the fault propagation 
information in the models. 

3. The alarm sequences are "filtered" according to a given sensor allocation plan. The 
filtered alarm sequence represents the primary alarms as received by the diagnostic 
system. 

4. Based on the primary alarm set, the diagnostic system analyzes the possible fault 
sources and fault causes. The resolution of the analysis depends on the actual 
composition of the primary alarm sources, i.e. the on the actual sensor allocation. 

5. The experimental results are evaluated and sensor allocation strategies are advised. 

We are currently at the modeling and model refinement phase of the study. The 
actual results of this activity are shown in the next section. 

Process and Fault Modeling for the ECLSS 

The starting point for this case study was an informal description of the ECLSS in 
terms of a layered process component graphic and a fault diagnosis handbook indicating 
possible faults of the system or of its subcomponents. The system is currently under develop- 
ment, so we are only able to show partial results. The main goal is to show a snapshot of 
how the ECLSS is represented and how our technology is used to obtain meaningfull 
problem representations of an overall process to be used for a variety of applications like 
fault diagnosis. 

Hierarchical Process Model (HPM) of the ECLSS 

The first step is to obtain a decomposition of the overall system. The decomposition 
does not necessarily follow the physical layout but rather the functional layout. The step- 
wise refinement of the ECLSS leads into the process hierarchy tree shown in Figure 7. 
Each node represents a certain function of the system. The function maps the specified 
input signals to the specified output signals. Signal dependencies in the hierarchy tree are 
only possible between father and son nodes or between sibling nodes. This guarantees 
complete modular development and any desired design direction. Any signal dependency is 
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Figure 7. Process Hierarchy of the ECLSS 
indicated by a connection. Connections usually describe a material flow. 

In the following, the process hierarchy is described in more detail. The top level 
process is the ECLSS process as shown in Picture 14. Since this process is currently not 
integrated in any other process it does not have any input and output signals. An integra- 
tion of the ECLSS later into the entire Space Station process would include an extension of 
the interface including e.g. electricity as a supply signal. The functional decomposition of 
the ECLSS system currently consists of three subprocesses: 


- Air Control 


- Potable Water Processing 

- Hygiene Processing 

On this level of the decomposition only water flow is considered between the sub- 
processes of the ECLSS. As Picture 14 shows the water flow itself forms a closed loop. The 
air control consumes water in the form of a supply from the hygiene water for the oxygen 
generation. How the water is used is hidden in the air control process itself and will be 
specified on a lower level of the decomposition. A second water supply for the air control is 
vapor contained in the air itself. The source of the vapor are module elements like dish 
washers and showers. These modules are modeled in the hygiene processing branch and 
therefore the vapor is an output of the hygiene processing. The air control itself supplies 
the potable water processing with two water sources, one as an outcome of the dehumdifica- 
tion and one as an outcome of the carbon dioxide processing both modeled in the air 
control process. This is explained below. The potable Water processing supplies the 
hygiene process with make-up water and with drinking water which will be consumed in 
subprocesses of the hygiene process. 


1. Air Control 

The air control is a more complex process model. Its decomposition is shown 
in Picture 15.a. The following subprocesses are modeling the functionality of the air 
control: 

- Temperature and Humidity Control (THC) 

- Ventilation 

- Oxygen Generation 

- Carbon dioxide Processing 

The interface signals of the air control as already used on the upper level of 
the process hierarchy, are water and vapor as input signals and dehumidification 
water and carbon dioxide reduction water as output signals. The air control basically 
models the water flow through the air and the air flow itself. The air flow itself is a 
closed loop, starting with the ventilation subprocess which pumps mixed air to the 






temperature and humidity control. The dehumdification of the THC generates 
water which is supplied through the output signal to connected processes of the air 
control. Further outcomes of the THC are air which is circulated by the ventilation 
system. A part of the air is used for the C02 processing and is designed as a 
separate signal. The C02 processing removes the C02 from this air in order to keep 
the desired C02 concentration in a certain range. The C02 reduced air is passed 
back to the ventilation. Further input signals of the ventilation are oxygen and 
hydrogen supplied by the oxygen processing and the vapor supplied through the 
external interface of the air control. The ventilation system basically mixes all of its 
input to a regular air composition. The oxygen processing generates hydrogen and 
oxygen out of the external water supply using an electrolysis process. The main part 
of the produced hydrogen is used by the C02 processing. 

The C02 processing maps its input signals, air and hydrogen, to its output 
signals water, hydrogen and carbon using two subprocesses. This is shown in Picture 
15.b. The C02 removal process extracts C02 from the air and supplies the C02 
reductions process. The C02 and the H2 are transformed with a Bosch process into 
H20 and Carbon. 


2. Potable Water Processing 

The second subprocess of the ECLSS is the potable water processing. The 
physical layout of this process includes two rows of each 4 tanks. At a specific time 
each tank has a certain unique functionality as filling, use, test or supply. While one 
of the tank rows is collecting the potable water from the air control process the 
other row is processing the water. Whenever one tank is full the tanks will switch 
their functionality i.e tested water can be used. Since we are more concerned about 
the functional decomposition rather than the physical decomposition we obtained 
the following subprocesses (shown in Picture 16). 

- Pumping 

- Distribution and Storage 

- Multi Filtration 


- Potable Water Testing 



ORIGINAL PAGE 
COLOR PHOTOGRAPH 



Pictures 15a-b: Process Models of the Air Control and C02 Processing subsystems 
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- Controller 


For the functionality of the potable water processing it is not important which 
tank is used for the filtration or for the water testing. This leads to the fact that on 
this level of decomposition the physical layout is completely ignored. However the 
physical layout is important for the entire model and can be modeled as a different 
view of the system. 

The pumping process takes the inputs dehumidification water and carbon 
reduction water and pumps it into the distribution and storage process. The distribu- 
tion and storage process basically models on a lower level of the hierarchy the tanks 
and their current state, respectively. The switching of the functionality is controlled 
by a controller process which receives status signals from the distribution process, 
and using a finite state automaton, control signals are generated to control pipe 
switches. Status signals are, for instance, the level of the tanks. The multi filtration 
process consumes water from the distribution process, filters it, and returns it back. 
The same applies to the potable water testing. 


3. Hygiene Processing 

The outcome of the distribution process is the supply of drinking water and 
make-up water exported through the interface and used on an upper level by the 
hygiene process. 

The overall task of the hygiene process is to supply modules like shower and 
dishwasher with waste water and to clean this water for further use. Furthermore 
the urine of the crew members has to be filtered in order not to loose water in the 
closed water loop of the ECLSS. 

The decomposition of the hygiene processing is shown in Picture 17.a. The 
central process is the hygiene subprocess. It supplies the user modules modeled in 
the subprocess circulation/consumer with filtered waste water and takes the col- 
lected water from this subprocess for its task as an input. Further input signals are 
body waste of the crew, processed urine and the makeup water supplied by the 
potable water processing. The output are the before mentioned waste water supply, 
clean water exported by the hygiene processing to the air control, and brine. 



original page 

COLOR PHOTOGRAPH 



Picture 16 : Incomplete Process Model of the Potable Water Processing subsystem 


The urine processing seperates and filters the urine and the brine and supplies 
the hygiene process with preprocessed urine. A further output of the hygiene 
processing is the vapor generated by the user modules. 

The waste water circulation is the only subprocess of the hygiene processing 
defined in more detail thus far (see Picture 17.b). It takes the supplied unused waste 
water, stores it, and distributes it to the different users. This function is modeled in 
the supprocess Waste Water Supply. Each User is modeled as a subprocess consum- 
ing the supplied waste water and returning used waste water and vapor. The vapor is 
actually contained in the air, but since the air flow is modeled in the air control, only 
the vapor has to be established as a separate signal. Waste water and vapor are 
collected by the supprocesses waste water collector and vapor addition and supplied 
through the interface. 

Declarative Form of the HPM 

Thus far the funtional structure and decomposition of the ECLSS has been described. 
The Graphic Automation Testbed facilitates the modeling process by a graphical editor. 
The use of graphic editors helps to avoid errors and increases the visibility of the system 
structure. For further use of the process model a more formalized representation has to be 
generated. This is done automatically by our system. Figure 8 shows the declarative form of 
the air control process. The declarative form contains all the necessary information to 
reconstruct the model and the graphical presentation of it. Furthermore the declarative 
form can be extended to carry more information about different views of the system like 
the fault modeling aspect. 

In a leaf in the process hierarchy only the input and output signals are specified. 
Pictures 18.a-b show an example for such a process. The graphical editor allows, at any 
time, extension of the process model to greater detail or modification of a process as long 
as the interface to process remains the same or is extended. The pictures show some pop 
up menus to extend a process model. It allows the user to specify new signals, subprocesses 
and connections between them. In order to represent subprocesses graphically the user can 
use a bit map editor to create an icon for a specific process. Each icon also contains a 
connection point for each input and output signal in order to enable the user to specify 
connections to and from those signals on a higher level. 
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Pictures 17a-b: Process Models of the Hygiene Processing and Waste Water Circula- 
tion subsystems 
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Figure 8. Declarative form of the Air Control Process 
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Once the process model is defined, different views of the processes can be considered 
and modeled. In this section we are considering a fault model for the ECLSS. We would 
like to emphasize that the fault specification is incomplete and needs further refinements 
to be used for diagnosis. 
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Pictures 18a-b: Interactive, graphical modeling and iconification of a Process 


The construction of a fault model is performed the same way as that of of a process 
model, namely using an appropriate graphical editor. Each fault model is represented on 
the higher level of the hierarchy by an icon where each icon has several connection points 
related to the failure modes of the subprocess. An important aspect of fault modeling is its 
close relationship to the process models. When a fault model for a process is to be defined, 
the fault model will inherit the basic structure from the HPM: (1) the name of the sub- 
processes, and (2) the causal links among the the subprocesses which is derived from the 
existence of physical links. This relationship guarantees that the HPM and HFM will be 
consistent. 

In the following, some of the failure modes and their relationships will be explained 
from the point of view of the diagnosis. The failure model of the ECLSS as shown in 
Picture 19. does not have any failure modes but defines possible relationships of failure 
modes of its submodels. For instance, if the failure mode of the potable water submodel is 
set, this can be due to the fact that there is an internal propagation inside the potable water 
process or due to the fact that the humidity is high or no water is produced by the air 
control process. 

1 . Failure Model of the Air Control 

Suppose that the system detects that their is no water supply for the potable 
water processing. Picture 20.a shows the failure model of the air control. The diag- 
nosis traces the error back to the ’No_water_for_potable_supply’ failure mode of 
the air control. The diagnosis tries to find an explanation by exploring the fault 
model for the air control. Dependent on whether the corresponding failure modes 
are set the THC model, C02 model and the 02 model are examined to find more 
detailed explanation of why the failure mode is set. Suppose the reduction error 
failure mode of the C02 process is not set, then there is no reason to check the C02 
process or the 02 process for more detail. The source of failure must be the THC 
module. 

Picture 20.b shows the failure model of the C02 processing. An reduction 
error might be caused by a failing of either one of the two sub processes. Both are 
due to check when the upper failure mode is set. 

2. Fault Model of the Potable Water Processing 

Picture 21 shows the fault model for the potable water processing. Beside the 



definition of the failure modes the model is incomplete. The graphical editor auto- 
matically displays the icons and the names of the submodels. Since the icons are not 
yet defined, a standard icon is shown to remind the user of the incompleteness of his 
system. 

3. Fault Model of the Hygiene Processing 

The fault model of the hygiene processing is shown in Picture 22.a. Two major 
failures are identified in the model: the water quality and the water quantity. The 
fact that the water is dirty might be due to the fact that the inverse osmosis of the 
central hygiene process is out of order or that the urine process fails or that the 
waste water quality is so low that the filtration of the hygiene process is overloaded. 
Low water supply can be traced back usually to a leakage in one of the subcom- 
ponents. 

Picture 22.b shows the incomplete fault model of the waste water circulation. 
Here some alarms are edited but not yet connected. Connections will cause the 
connected failure modes to be activated whenever the alarms go on. 

As mentioned earlier the fault model is one aspect of a process model besides the 
structural view. Therefore the fault model is also stored in the declarative form of the 
process model. This is shown in Figure 9 for the hygiene process. The declarative form 
holds structural and fault model information about the process in different view slots, 
which are both accessed by the diagnosis system. 

Testing of P iagnosabilitv 

After completing the HPM and HFM for the ECLSS, the remaining steps of the 
analysis will be executed. 

1. First, an alarm detection model will be defined which associates each failure mode 
with a separate alarm. 

2. The real-time alarm pattern simulator will be generated by using this model, i.e. an 
introduced fault will generate a complete alarm sequence. The diagnostic system 
will obviously identify the exact failure source, because of the highly redundant 
failure mode detection. 
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Picture 19: Failure Model of ECLSS 
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Pictures 20a-b: Failure Models of the Air Control and C02 Processing subsystems 
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Picture 21: Failure Model of the Potable Water Processing subsystem 
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Figure 9. Fault Diagnosis Declarative Form of the Higiene Process 

By introducing "filters" in the alarm sequence, we will simulate situations with less 
primary alarm sources, i.e. with less sensors. The output of the diagnostic process 
will reflect the effect of the decrease of the redundancy in terms of decreasing 
resolution of the diagnosis. 
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Pictures 22a-b: Failure Models of the Hygiene Processing and Waste Water Circula- 
tion subsystems 
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5. FUTURE WORK 

In the first year of the project we have obtained experiences with the potential of a 
graphic workstation environment in robot simulation and have tested the new capabilities 
of the AI extension of ROBOSIM. The research has been conducted in three parallel 
directions: 

1. Improvement of the basic capabilities of ROBOSIM by taking advantage of the 
graphic workstation environment. 

2. Integration of ROBOSIM with Al-based modeling techniques and extension of the 
capabilities of the basic system with support toward general automation problems. 

3. Continuous testing of the system with a variety of application problems. 

In the next performance period we plan to continue these activities. The specific tasks 
that we will work on are the following: 

1. Improvement of the basic capabilities of ROBOSIM: We will continue working on 
the implementation of a fast inverse kinematic algorithm, on the improvement of 
the collision detection system, and on the extension of the basic package with 
dynamics. 

2. Integration of ROBOSIM with Al-based techniques: We will continue the im- 
plementation of the new modeling and representation techniques described in detail 
in Section 2. This new, Al-based representation system will make it possible to 
integrate ROBOSIM with other Al-based packages, such as the NASA-developed 
CLIPS system and the MULTIGRAPH system. The integration will ensure that 
ROBOSIM can serve as a general purpose robotics and automation simulation 
package. 

3. Application systems: We plan to continue testing the ROBOSIM package in dif- 
ferent application problems related to Space Station automation and robotics. The 
application experiences provide valuable feedback for the further improvement of 
the system. 
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Going a step beyond of these specific tasks, there is an other important area of 
development possibilities. Currently many companies in the aerospace community are 
developing various testbeds for automation systems. (For example the SSFP system 
developed by the Ml l KA corporation.) Many of these testbeds are targeted for designing 
and testing different subsystems of space-based automation systems (for example the 
automation systems of the Space Station). We think that it would be advantageous if data 
could be shared between the different testbeds, since it would allow a more comprehensive 
testing of these automation systems, than any single testbed alone. That is why it is our 
intention to study the possibility of developing interface packages which would enable us to 
use data from other systems in our automation and robotics testbed. 


